[00:27:23] 10DBA, 10Data-Services, 10Goal, 10cloud-services-team (FY2017-18): Migrate all users to new Wiki Replica cluster and decommission old hardware - https://phabricator.wikimedia.org/T142807#3695569 (10bd808) [00:33:34] 10DBA, 10Data-Services, 10Goal, 10cloud-services-team (FY2017-18): Migrate all users to new Wiki Replica cluster and decommission old hardware - https://phabricator.wikimedia.org/T142807#3695570 (10bd808) [00:49:03] 10DBA, 10Data-Services, 10Goal, 10cloud-services-team (FY2017-18): Migrate all users to new Wiki Replica cluster and decommission old hardware - https://phabricator.wikimedia.org/T142807#3695588 (10bd808) [00:49:05] 10DBA, 10Data-Services, 10User-bd808, 10cloud-services-team (Kanban): Create and announce timeline for shutting down labsdb100[13] - https://phabricator.wikimedia.org/T175086#3695586 (10bd808) 05Open>03Resolved Announced at https://lists.wikimedia.org/pipermail/cloud-announce/2017-October/000005.html [03:13:27] 10DBA, 10MediaWiki-extensions-WikibaseClient, 10Wikidata, 10Chinese-Sites: Enable description usage tracking on further wikis - https://phabricator.wikimedia.org/T178515#3695606 (10Shizhao) [05:27:17] 10Blocked-on-schema-change, 10DBA, 10Patch-For-Review: Drop now redundant indexes from pagelinks and templatelinks - https://phabricator.wikimedia.org/T174509#3695635 (10Marostegui) [05:28:04] 10Blocked-on-schema-change, 10DBA, 10Patch-For-Review: Drop now redundant indexes from pagelinks and templatelinks - https://phabricator.wikimedia.org/T174509#3682389 (10Marostegui) [05:28:27] 10Blocked-on-schema-change, 10DBA, 10Patch-For-Review: Drop now redundant indexes from pagelinks and templatelinks - https://phabricator.wikimedia.org/T174509#3683468 (10Marostegui) [05:54:36] 10DBA, 10MediaWiki-extensions-WikibaseClient, 10Wikidata, 10Chinese-Sites: Enable description usage tracking on further wikis - https://phabricator.wikimedia.org/T178515#3695654 (10Marostegui) That is fine by me - let's leave ruwiki aside for now while we finish purging up its recentchanges table (T177772)... [06:07:54] 10DBA, 10Patch-For-Review: Productionize 22 new codfw database servers - https://phabricator.wikimedia.org/T170662#3695658 (10Marostegui) [06:32:35] 10DBA, 10Patch-For-Review: Support multi-instance on core hosts - https://phabricator.wikimedia.org/T178359#3695667 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by marostegui on neodymium.eqiad.wmnet for hosts: ``` db2092.codfw.wmnet ``` The log can be found in `/var/log/wmf-auto-reimage/2017101... [06:51:06] 10DBA, 10Patch-For-Review: Support multi-instance on core hosts - https://phabricator.wikimedia.org/T178359#3695677 (10ops-monitoring-bot) Completed auto-reimage of hosts: ``` ['db2092.codfw.wmnet'] ``` Of which those **FAILED**: ``` ['db2092.codfw.wmnet'] ``` [06:51:23] 10DBA, 10Patch-For-Review: Support multi-instance on core hosts - https://phabricator.wikimedia.org/T178359#3695678 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by marostegui on neodymium.eqiad.wmnet for hosts: ``` db2092.codfw.wmnet ``` The log can be found in `/var/log/wmf-auto-reimage/2017101... [06:51:24] 10DBA, 10Patch-For-Review: Support multi-instance on core hosts - https://phabricator.wikimedia.org/T178359#3695679 (10ops-monitoring-bot) Completed auto-reimage of hosts: ``` ['db2092.codfw.wmnet'] ``` Of which those **FAILED**: ``` ['db2092.codfw.wmnet'] ``` [06:53:43] did mydumper work? [06:53:44] 10DBA, 10Patch-For-Review: Support multi-instance on core hosts - https://phabricator.wikimedia.org/T178359#3695680 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by marostegui on neodymium.eqiad.wmnet for hosts: ``` db2092.codfw.wmnet ``` The log can be found in `/var/log/wmf-auto-reimage/2017101... [06:53:54] jynus: yep, working :-) [06:54:00] finishing the import [06:54:03] was it the version? [06:54:09] looks so [06:54:34] so might be woirth to backport 0.9 to jessie i believe [07:25:30] 10DBA, 10Patch-For-Review: Support multi-instance on core hosts - https://phabricator.wikimedia.org/T178359#3695708 (10ops-monitoring-bot) Completed auto-reimage of hosts: ``` ['db2092.codfw.wmnet'] ``` and were **ALL** successful. [07:49:11] 10DBA, 10Gerrit, 10Operations, 10Release-Engineering-Team, 10Security: Gerrit: Convert gerrit's db caractor encoding from utf8 to utf8mb4 to prevent truncation of astral characters - https://phabricator.wikimedia.org/T153899#3695742 (10Dzahn) [08:14:43] for some reason, dbstore1001 dumps take 1 day to run SELECT /*!40001 SQL_NO_CACHE */ * FROM `ip_changes` on enwiki [08:15:02] is that the new table? [08:15:10] the? [08:15:19] it is relatively recent [08:16:04] but it is only 3GB [08:17:00] maybe I should convert it to tokudb? [08:17:34] i was going to ask: is it toku? [08:17:43] because I was assuming maybe toku was doing something weird [08:17:49] it is compact [08:18:07] I converted wb_terms to toku [08:18:22] I think it is a combination of having some internal issue with the load [08:18:32] rebuilding them works [08:18:45] but it cannot take 3 days to do a weekly backup [08:18:53] no, that is mad [08:18:58] try rebuilding it [08:19:12] yeah, when the select finishes :-) [08:19:19] otherwise, metadata lock [08:21:09] dbstore2001 takes 17 hours for 6 replica sets [08:21:27] it is low on resources, but no major blockage [08:46:00] 10DBA, 10Patch-For-Review: Support multi-instance on core hosts - https://phabricator.wikimedia.org/T178359#3695949 (10Marostegui) [09:08:11] 10Blocked-on-schema-change, 10DBA, 10MediaWiki-extensions-ORES, 10MW-1.29-release (WMF-deploy-2017-04-25_(1.29.0-wmf.21)), and 5 others: Concerns about ores_classification table size on enwiki - https://phabricator.wikimedia.org/T159753#3696073 (10Ladsgroup) All tables in all wikis are in the cleanest stat... [09:11:42] going for a cofee, will return in 30 minutes; I left you homework [09:12:00] o/ [09:55:11] hello people! Is https://gerrit.wikimedia.org/r/#/c/384979 ok to apply the no-reimage trick for you [09:55:14] ?? [10:06:43] meeting [11:08:06] 10Blocked-on-schema-change, 10DBA, 10Patch-For-Review: Drop now redundant indexes from pagelinks and templatelinks - https://phabricator.wikimedia.org/T174509#3696325 (10Marostegui) [12:32:42] 10DBA, 10Patch-For-Review: Support multi-instance on core hosts - https://phabricator.wikimedia.org/T178359#3696468 (10Marostegui) [12:33:10] 10DBA, 10Patch-For-Review: Productionize 22 new codfw database servers - https://phabricator.wikimedia.org/T170662#3696469 (10Marostegui) [12:55:40] 10DBA, 10Patch-For-Review: Support multi-instance on core hosts - https://phabricator.wikimedia.org/T178359#3696519 (10Marostegui) [13:22:25] marostegui: I do not have your gpg key, I think [13:23:46] yousure? [13:23:55] jynus: you should, as part of pwstore [13:36:09] volans: s/having it/trust it/ [13:36:49] I thing for ops coming after it was setup, moritz and/or daniel handled them [13:37:08] the keys in pwstore must be signed by at least other 2 ops IIRC [13:37:18] yeah [13:37:26] yeah, I am not saying he has done anything wrong [13:37:30] i remember rob and mortizm maybe signed it for me [13:37:36] I just said I didn't [13:41:10] and saying sorry because I couldn't encrypt somthing with his key [13:42:03] :( [13:42:20] * marostegui goes to a corner [14:20:31] I have updated our pending/ongoing projects etherpad [14:21:12] in our sync etherpad you mean? [14:21:21] yeah [14:21:36] I will send an email if I have something else to say [14:21:50] ah I see it now [14:21:52] thanks [14:25:34] jynus: first attempt to refactor the analytics eventlogging code in https://gerrit.wikimedia.org/r/#/c/385173 [14:26:05] probably it is still far from what you imagined but if you have time during today/tomorrow it would be great to discuss how to proceed [14:26:07] elukey: I saw it earlier but didn't have time to check it in detail [14:26:13] sure sure! [14:26:19] I mentioned the idea of refactoring [14:26:27] you seem to have doen a lot of work already!" [14:26:32] thank you! [14:27:09] the naming probably will need to be adjusted but I hope that you'll like the idea :) [14:27:50] if the puppet part is sorted before you go on vacation I'll work with marostegui next week (if he has time) on db1108 [14:28:08] otherwise we can wait [14:29:22] It looks ok, modules/profile/manifests/mariadb/misc/eventlogging/database.pp is a bit peculiar, but I cannot imagine a beter alternative [14:29:39] maybe we can make eventlogging-db a top role? [14:30:04] I don't know it is ok as it is [14:30:49] lol at pure_replica.yaml too :-) [14:31:22] we should rename replica.yaml to impure_replica.yaml [14:31:27] :-) juist kidding [14:48:46] db2092 low on space? [14:49:43] i am compressing innodb there [14:49:45] I see [14:49:49] it is one of the new multi-instance [14:49:54] I was surprised [14:49:57] thank you [14:50:07] yeah [14:50:21] it should be probably half of that once compressed [14:50:35] plus the compression needs even more space [15:05:56] hahahah yes pure_replica is horrible but I didn't find a better name :P [15:08:06] 10DBA, 10Analytics, 10Operations: Prep to decommission old dbstore hosts (db1046, db1047) - https://phabricator.wikimedia.org/T156844#3696790 (10jcrespo) [15:59:36] 10DBA, 10Analytics: Access to x1 broken on stat1006 - https://phabricator.wikimedia.org/T178237#3685563 (10elukey) Yes we switched the CNAME on purpose a while ago (https://gerrit.wikimedia.org/r/#/c/378211/1/templates/wmnet) to avoid access from the research user to db1029. We could set up on dbstore1002 the... [16:02:44] 10DBA, 10Analytics: Access to x1 broken on stat1006 - https://phabricator.wikimedia.org/T178237#3685563 (10jcrespo) This is a duplicate. The problem is it is not easy to replicate x1 because it duplicates db names (e.g. enwiki and enwiki are both on s1 and x1). I promised to provide temporary access soon, if I... [16:03:45] 10DBA, 10Operations: Lost access to x1-analytics-slave - https://phabricator.wikimedia.org/T175970#3696906 (10jcrespo) [16:03:47] 10DBA, 10Analytics: Access to x1 broken on stat1006 - https://phabricator.wikimedia.org/T178237#3696908 (10jcrespo) [16:04:44] 10DBA, 10Operations: Lost access to x1-analytics-slave - https://phabricator.wikimedia.org/T175970#3609511 (10jcrespo) CC @Marostegui Maybe we can setup an unpuppetized copy of x1 from dbstore2002 on dbstore1002? [21:28:17] 10Blocked-on-schema-change, 10DBA, 10Data-Services, 10Dumps-Generation, 10MediaWiki-Platform-Team (MWPT-Q2-Oct-Dec-2017): Schema change for refactored comment storage - https://phabricator.wikimedia.org/T174569#3698106 (10CCicalese_WMF)