[01:39:17] 10DBA, 10AbuseFilter, 10MediaWiki-Change-tagging: Cannot add a previously used change tag to an abuse filter - https://phabricator.wikimedia.org/T173917#4148598 (10TTO) Let's just take a step back here: what problem are we trying to solve here? It seems that we need to allow tags used by other abuse filters,... [05:29:22] 10DBA, 10Epic, 10Tracking: Database tables to be dropped on Wikimedia wikis and other WMF databases (tracking) - https://phabricator.wikimedia.org/T54921#4148681 (10Marostegui) [05:29:27] 10DBA, 10Collaboration-Team-Triage, 10StructuredDiscussions, 10Patch-For-Review, 10Schema-change: Drop flow_subscription table - https://phabricator.wikimedia.org/T149936#4148679 (10Marostegui) 05Open>03Resolved I have dropped this table on s3 primary master. It was an empty table. [06:00:16] 10DBA, 10AbuseFilter, 10MediaWiki-Change-tagging: Cannot add a previously used change tag to an abuse filter - https://phabricator.wikimedia.org/T173917#4148689 (10Daimona) @TTO Not exactly. I agree until (3.), but IMHO there's another check (to be implemented in 2.): - Was the tag ever used by an abuse fi... [06:16:03] 10DBA, 10AbuseFilter, 10MediaWiki-Change-tagging: Cannot add a previously used change tag to an abuse filter - https://phabricator.wikimedia.org/T173917#4148691 (10TTO) >>! In T173917#4148689, @Daimona wrote: > - Was the tag ever used by an abuse filter? If so, the tag is OK. Is that really needed, though... [06:18:35] 10DBA: Drop backup tables for old schema changes (aka "x_pre_y" tables) - https://phabricator.wikimedia.org/T118859#4148693 (10Marostegui) The tables are backuped at: `dbstore1001:/srv/backups/tmp/T118859` [06:24:30] 10DBA: Drop backup tables for old schema changes (aka "x_pre_y" tables) - https://phabricator.wikimedia.org/T118859#4148694 (10Marostegui) [06:24:47] 10DBA, 10AbuseFilter, 10MediaWiki-Change-tagging: Cannot add a previously used change tag to an abuse filter - https://phabricator.wikimedia.org/T173917#4148695 (10Daimona) @TTO Yes, it's the main problem for which this task was created :-) However, I didn't clearly understand your point (3.) at first. I jus... [06:33:45] 10DBA: Drop backup tables for old schema changes (aka "x_pre_y" tables) - https://phabricator.wikimedia.org/T118859#4148700 (10Marostegui) [06:38:05] 10DBA, 10AbuseFilter, 10MediaWiki-Change-tagging: Cannot add a previously used change tag to an abuse filter - https://phabricator.wikimedia.org/T173917#4148701 (10TTO) >>! In T173917#4148695, @Daimona wrote: > @TTO Yes, it's the main problem for which this task was created :-) I meant a scenario in my new... [07:14:24] 10DBA, 10AbuseFilter, 10MediaWiki-Change-tagging: Cannot add a previously used change tag to an abuse filter - https://phabricator.wikimedia.org/T173917#4148747 (10Daimona) @TTO Indeed :-) However, I changed my mind again. We're currently switching AF to OOUI, and in the editing interface we'd like to provid... [07:47:48] 10DBA: Drop backup tables for old schema changes (aka "x_pre_y" tables) - https://phabricator.wikimedia.org/T118859#4148815 (10Marostegui) [07:58:16] 10DBA, 10AbuseFilter, 10MediaWiki-Change-tagging: Cannot add a previously used change tag to an abuse filter - https://phabricator.wikimedia.org/T173917#3544310 (10jcrespo) > Adding DBA since we're adding a table and need approval. > If the DBAs are happy to allow the queries in fetchAllTags to be run more... [08:11:54] 10DBA: Drop backup tables for old schema changes (aka "x_pre_y" tables) - https://phabricator.wikimedia.org/T118859#4148843 (10Marostegui) [08:27:45] 10DBA: Drop backup tables for old schema changes (aka "x_pre_y" tables) - https://phabricator.wikimedia.org/T118859#4148894 (10Marostegui) [08:30:25] 10DBA: Drop backup tables for old schema changes (aka "x_pre_y" tables) - https://phabricator.wikimedia.org/T118859#4148901 (10Marostegui) [08:37:47] 10DBA, 10AbuseFilter, 10MediaWiki-Change-tagging: Cannot add a previously used change tag to an abuse filter - https://phabricator.wikimedia.org/T173917#4148905 (10Daimona) @jcrespo we don't have any code yet, but I'll send my patch later in the afternoon. Then we'll have to discuss about the approach, see i... [08:46:52] 10DBA: Drop backup tables for old schema changes (aka "x_pre_y" tables) - https://phabricator.wikimedia.org/T118859#4148925 (10Marostegui) [08:48:30] 10DBA: Drop backup tables for old schema changes (aka "x_pre_y" tables) - https://phabricator.wikimedia.org/T118859#4148932 (10Marostegui) [08:48:45] 10DBA: Drop backup tables for old schema changes (aka "x_pre_y" tables) - https://phabricator.wikimedia.org/T118859#1811055 (10Marostegui) 05Open>03Resolved a:03Marostegui All done. Backups are at: T118859#4148693 [08:48:48] 10DBA, 10Epic, 10Tracking: Database tables to be dropped on Wikimedia wikis and other WMF databases (tracking) - https://phabricator.wikimedia.org/T54921#4148936 (10Marostegui) [08:54:39] 10Blocked-on-schema-change, 10DBA, 10Patch-For-Review, 10User-Ladsgroup, 10Wikidata-Ministry-Of-Magic: Schema change for rc_namespace_title_timestamp index - https://phabricator.wikimedia.org/T191519#4148946 (10Marostegui) [08:54:58] 10Blocked-on-schema-change, 10DBA, 10Data-Services, 10MediaWiki-Platform-Team (MWPT-Q4-Apr-Jun-2018), 10Patch-For-Review: Schema change for refactored actor storage - https://phabricator.wikimedia.org/T188299#4148947 (10Marostegui) [08:55:19] 10Blocked-on-schema-change, 10DBA, 10Multi-Content-Revisions, 10Patch-For-Review, 10User-Addshore: Change DEFAULT 0 for rev_text_id on production DBs - https://phabricator.wikimedia.org/T190148#4148948 (10Marostegui) [09:32:48] the patch applied means there will be no longer a /srv/{sqldata,tmp} on clients and multi-intstance hosts [09:33:05] cool, it sometimes causes confusion :) [09:34:35] the next one will take care of creating /run/mysqld on start, but that will take more time as it affects all hosts [09:46:06] <_joe_> hey just as a notification, as I don't expect problems: we're performing a rolling restart of the memcached hosts [09:46:18] <_joe_> if you see some increase in db activity, that might be the reason [09:46:25] thanks for the heads up [09:47:00] <_joe_> actually, this could be a chance to test our resilience losing one MC row, if we want to [09:47:18] <_joe_> if you don't have major maintenance ongoing on your side [09:47:39] We are doing some reimages in eqiad, and some alters on codfw (which shouldn't be a blocker) [09:47:54] <_joe_> codfw is unaffected [09:48:01] <_joe_> I restarted memcacheds earlier [09:49:36] <_joe_> so you would be ok with the test? It can put some strain on the databases, so I want your green light [09:50:03] can we ramp up slowly? [09:50:28] <_joe_> that was the original plan - do one server at a time, with a spacing of 1 hour between them [09:50:43] well, I thought you wanted to do more [09:50:45] <_joe_> but I just thought this could be a chance to test row-failure resiliency [09:50:48] I was just asking to do [09:50:58] 1 at the same time, then wait, then 2, etc. [09:51:05] <_joe_> which by definition means restarting N [09:51:37] so increasing slowly the number to do at the same time [09:51:45] <_joe_> but we can just schedule this at another time [09:51:59] <_joe_> as elukey was suggesting elsewhere, it would be nice to involve performance too [09:52:15] <_joe_> so yeah, let's not do it this time [10:17:36] jynus: marostegui hey, I already added you in the gerrit patch and when you want to deploy it just let me know to check :) thank you! https://gerrit.wikimedia.org/r/#/c/428297/ [10:20:57] meeting :) [10:21:02] will ping you later [10:27:58] cool [10:28:07] thanks! [12:00:58] 10DBA, 10AbuseFilter, 10MediaWiki-Change-tagging: Cannot add a previously used change tag to an abuse filter - https://phabricator.wikimedia.org/T173917#4149427 (10TTO) >>! In T173917#4148747, @Daimona wrote: > With your method, this isn't possible (you cannot list every non-existent tag) Regardless of whic... [12:04:17] 10DBA, 10AbuseFilter, 10MediaWiki-Change-tagging: Cannot add a previously used change tag to an abuse filter - https://phabricator.wikimedia.org/T173917#4149430 (10TTO) @jcrespo One of the things we are talking about is the queries in [[https://phabricator.wikimedia.org/diffusion/EABF/browse/master/includes/... [12:16:08] 10DBA, 10AbuseFilter, 10MediaWiki-Change-tagging: Cannot add a previously used change tag to an abuse filter - https://phabricator.wikimedia.org/T173917#4149478 (10Daimona) @TTO Right. What if we intersect such dedicated table with the list of tags in special:tags? Unused tags would still be included and if... [12:24:07] 10DBA, 10AbuseFilter, 10MediaWiki-Change-tagging: Cannot add a previously used change tag to an abuse filter - https://phabricator.wikimedia.org/T173917#4149530 (10TTO) If you code it up I will be able to let you know whether it makes sense :) [13:01:00] 10Blocked-on-schema-change, 10DBA, 10Patch-For-Review, 10User-Ladsgroup, 10Wikidata-Ministry-Of-Magic: Schema change for rc_namespace_title_timestamp index - https://phabricator.wikimedia.org/T191519#4149628 (10Marostegui) s6 eqiad progress: [] labsdb1009 [] labsdb1010 [] labsdb1011 [] db1102 [] dbstore... [13:01:18] 10Blocked-on-schema-change, 10DBA, 10Data-Services, 10MediaWiki-Platform-Team (MWPT-Q4-Apr-Jun-2018), 10Patch-For-Review: Schema change for refactored actor storage - https://phabricator.wikimedia.org/T188299#4149629 (10Marostegui) s6 eqiad progress: [] labsdb1009 [] labsdb1010 [] labsdb1011 [] db1102 [... [13:01:36] 10Blocked-on-schema-change, 10DBA, 10Multi-Content-Revisions, 10Patch-For-Review, 10User-Addshore: Change DEFAULT 0 for rev_text_id on production DBs - https://phabricator.wikimedia.org/T190148#4149630 (10Marostegui) s6 eqiad progress: [] labsdb1009 [] labsdb1010 [] labsdb1011 [] db1102 [] dbstore1002 [... [13:06:08] 10Blocked-on-schema-change, 10DBA, 10Patch-For-Review, 10User-Ladsgroup, 10Wikidata-Ministry-Of-Magic: Schema change for rc_namespace_title_timestamp index - https://phabricator.wikimedia.org/T191519#4149653 (10Marostegui) [13:06:22] 10Blocked-on-schema-change, 10DBA, 10Data-Services, 10MediaWiki-Platform-Team (MWPT-Q4-Apr-Jun-2018), 10Patch-For-Review: Schema change for refactored actor storage - https://phabricator.wikimedia.org/T188299#4149654 (10Marostegui) [13:06:45] 10Blocked-on-schema-change, 10DBA, 10Multi-Content-Revisions, 10Patch-For-Review, 10User-Addshore: Change DEFAULT 0 for rev_text_id on production DBs - https://phabricator.wikimedia.org/T190148#4149655 (10Marostegui) [13:13:50] 10DBA, 10Operations: Investigate dropping "edit_page_tracking" database table from Wikimedia wikis after archiving it - https://phabricator.wikimedia.org/T57385#4149677 (10Marostegui) @Nemo_bis ^ [15:32:46] do you know why there is lag on labsdb1011, and long-running queries? [15:32:57] yeah, i checked earlier [15:33:04] there were lots of long queries [15:33:13] Some of them have been already killed by the query killer [15:48:13] 10Blocked-on-schema-change, 10DBA, 10Patch-For-Review, 10User-Ladsgroup, 10Wikidata-Ministry-Of-Magic: Schema change for rc_namespace_title_timestamp index - https://phabricator.wikimedia.org/T191519#4150237 (10Marostegui) [15:48:30] 10Blocked-on-schema-change, 10DBA, 10Data-Services, 10MediaWiki-Platform-Team (MWPT-Q4-Apr-Jun-2018), 10Patch-For-Review: Schema change for refactored actor storage - https://phabricator.wikimedia.org/T188299#4150238 (10Marostegui) [15:48:35] 10Blocked-on-schema-change, 10DBA, 10Multi-Content-Revisions, 10Patch-For-Review, 10User-Addshore: Change DEFAULT 0 for rev_text_id on production DBs - https://phabricator.wikimedia.org/T190148#4150239 (10Marostegui) [15:50:23] 10DBA, 10Operations, 10ops-eqiad, 10Patch-For-Review: Rack and setup db1116 - db1123 - https://phabricator.wikimedia.org/T191792#4150253 (10Cmjohnson) [15:51:45] 10DBA, 10Operations, 10ops-eqiad, 10Patch-For-Review: Rack and setup db1116 - db1123 - https://phabricator.wikimedia.org/T191792#4116638 (10Cmjohnson) @Marostegui @jynus All 8 of the db's are racked, cabled and idrac setup. Updated racktables. These should be ready for you later today/first thing tomorro... [15:52:34] 10DBA, 10Operations, 10ops-eqiad, 10Patch-For-Review: Rack and setup db1116 - db1123 - https://phabricator.wikimedia.org/T191792#4150257 (10Marostegui) @Cmjohnson thanks a lot - RAID also set up? [15:59:47] did you depool labsdb1010? it seems as if it had been depooled [16:00:17] with high connection problems https://grafana.wikimedia.org/dashboard/db/mysql?orgId=1&from=now-1h&to=now&var-dc=eqiad%20prometheus%2Fops&var-server=labsdb1010&var-port=9104 [16:01:50] jynus: I did, I !log'ed it early in the morning [16:02:08] no, I mean some minutes ago [16:02:15] I repooled it [16:02:30] the connection problems started minutes ago [16:02:54] maybe an spike when it got repooled? [16:03:05] qps is going down [16:03:34] maybe an initial spike when it got repooled [16:05:06] ok, it may be that, I think it is going back to normal levels now [16:05:34] yeah, still seeing as Up in the proxy [16:05:56] it got a concerning spike of errors just minutes ago [16:06:19] maybe we can start repooling them slower like we do with core hosts [16:06:25] like with smaller weight or something [18:52:38] 10DBA, 10Cloud-Services, 10User-Urbanecm, 10cloud-services-team (Kanban): Prepare and check storage layer for inhwiki - https://phabricator.wikimedia.org/T184375#4151037 (10Bstorm) a:03Bstorm [18:52:53] 10DBA, 10Cloud-Services, 10User-Urbanecm, 10cloud-services-team (Kanban): Prepare and check storage layer for inhwiki - https://phabricator.wikimedia.org/T184375#3881370 (10Bstorm) I'll take care of this while I'm adding some new views. [18:54:16] 10DBA, 10Cloud-Services, 10User-Urbanecm, 10cloud-services-team (Kanban): Prepare and check storage layer for romdwikimedia - https://phabricator.wikimedia.org/T187774#4151046 (10Bstorm) a:03Bstorm [18:57:41] 10DBA, 10Cloud-Services, 10Hindi-Sites, 10User-Jayprakash12345, 10cloud-services-team (Kanban): Prepare and check storage layer for hiwikimedia - https://phabricator.wikimedia.org/T188490#4151057 (10Bstorm) a:03Bstorm [18:58:17] 10DBA, 10Cloud-Services, 10User-Urbanecm, 10cloud-services-team (Kanban): Prepare and check storage layer for gorwiki - https://phabricator.wikimedia.org/T189112#4151058 (10Bstorm) a:03Bstorm [18:58:42] 10DBA, 10Cloud-Services, 10User-Urbanecm, 10cloud-services-team (Kanban): Prepare and check storage layer for gorwiki - https://phabricator.wikimedia.org/T189112#4031710 (10Bstorm) Grabbing these view creates because I'm messing with some things on the labsdb servers anyway. [18:59:12] 10DBA, 10Cloud-Services, 10cloud-services-team (Kanban): Prepare storage layer for euwikisource - https://phabricator.wikimedia.org/T189466#4151060 (10Bstorm) a:03Bstorm [18:59:31] 10DBA, 10Cloud-Services, 10User-Urbanecm, 10cloud-services-team (Kanban): Prepare storage layer for lfnwiki - https://phabricator.wikimedia.org/T183566#4151061 (10Bstorm) a:03Bstorm [19:08:16] 10DBA, 10AbuseFilter, 10MediaWiki-Change-tagging, 10Patch-For-Review: Cannot add a previously used change tag to an abuse filter - https://phabricator.wikimedia.org/T173917#4151087 (10Daimona) @TTO There we go :) My thoughts are summarized in commit message. The patch itself is still quite WIPpy, but I dec...