[00:15:21] no_justification: is ActiveAbstract wfLoadExtension()'d in LocalSettings.php? [00:15:30] cause if not, I see exactly why it fails [00:15:54] the plugin isn't registered, and nothing sets up an autoloader for it [00:20:13] 10Release-Engineering-Team (Kanban), 10Operations, 10Release Pipeline: Package/upload service-checker for Debian stretch - https://phabricator.wikimedia.org/T184224#3885164 (10dduvall) @Joe, will the existing service-checker package for jessie work for stretch? The dependencies don't look all that complicated. [00:35:04] PROBLEM - Puppet errors on deployment-cache-text04 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [00:47:19] 10Gerrit: Migrate to NoteDb - https://phabricator.wikimedia.org/T174034#3885229 (10Dzahn) >>! In T174034#3884521, @Paladox wrote: > Comments are now stored in the repo so it store the history. Cool, i was always worried that we might lose all the comment history if we ever were to use a different code review t... [00:49:47] brion: Ummmm...... [00:50:25] That's a question for dumps....it's not loaded in CommonSettings at all.... [00:51:26] Yeah my suspicion is we never added it to the config because the --plugin loading used to be enough [00:51:45] It should be harmless to toss it into CommonSettkngs.php [00:52:01] Then can remove the explicit path [00:55:42] That seems like it'd work [00:56:35] Could probably limit it to PHP_SAPI == 'cli' [00:56:56] So we can avoid loading a pointless extension for web requests [00:57:34] Yep [01:05:38] still not in the bed so I'll hang out a tiny bit longer in here [01:06:25] Hey apergos we think we solved that dump issue [01:06:55] I saw the comment [01:06:59] sure seems likely [01:06:59] Due to the changes in the extensions layout it needs to actually be wfLoadExtensioned [01:07:02] Yay [01:07:22] I wonder about the side effects if any of adding it to the loaded extensions in CommonSettings like all the rest [01:07:34] is there anything that can be exploited now via the web that we would have to worry about? [01:07:44] Should be safe, but no_justification posited only loading it on CLI [01:08:01] if that's not hard to do, that would make me happier for sure [01:08:20] Yeah just check PHP_SAPI_NAME [01:09:06] seems we don't have any config settings where we do that now [01:09:07] interesting [01:13:14] some of these scripts check the value in order to bail if called from cli [01:13:26] not the abstract ones, I mean other extensions [01:15:07] Well in theory I guess you could extend the web export to support the plugins ;) but... Prolly not hehe [01:18:29] er nope ;-D [01:18:46] let's make this as easy as possible instead of as complicated as possible :-D :-D [01:22:37] what I do need is some clever format for wikidatawiki pages [01:22:49] which right now have an absolutely useless abstract as it turns out [01:23:46] I suppose a lot of the heavily templated wiktionaries must be like that too, would be nice to check. but wikidata is the sore thumb here [01:34:34] PROBLEM - Free space - all mounts on deployment-mx is CRITICAL: CRITICAL: deployment-prep.deployment-mx.diskspace._var_log.byte_percentfree (<100.00%) [02:04:08] right, gone for real ths time. cya! [02:33:41] 10Release-Engineering-Team, 10Language-Team, 10MediaWiki-extensions-WikimediaIncubator, 10Epic, 10I18n: Make creating a new Language project easier - https://phabricator.wikimedia.org/T165585#3885407 (10CCicalese_WMF) [04:53:12] 10Beta-Cluster-Infrastructure, 10Puppet: Puppet broken on deployment-eventlogging04 due to missing directory '/var/lib/superset'? - https://phabricator.wikimedia.org/T184238#3885503 (10mmodell) [05:02:41] 10Beta-Cluster-Infrastructure, 10ORES, 10Scoring-platform-team, 10Puppet: Puppet broken on deployment-ores01 due to missing hieradata - https://phabricator.wikimedia.org/T184478#3885510 (10Ladsgroup) Yes, That's an ongoing work which I failed to finish yesterday and this is the first thing to pick up and f... [05:36:08] PROBLEM - Free space - all mounts on deployment-fluorine02 is CRITICAL: CRITICAL: deployment-prep.deployment-fluorine02.diskspace._srv.byte_percentfree (<11.11%) [05:36:30] 10Beta-Cluster-Infrastructure, 10ORES, 10Patch-For-Review, 10Scoring-platform-team (Current), and 2 others: Move beta cluster ORES to its own machine - https://phabricator.wikimedia.org/T184282#3885538 (10Ladsgroup) a:03Ladsgroup [05:36:41] RECOVERY - Puppet errors on deployment-ores01 is OK: OK: Less than 1.00% above the threshold [0.0] [05:36:43] PROBLEM - Mediawiki Error Rate on graphite-labs is CRITICAL: CRITICAL: 80.00% of data above the critical threshold [10.0] [05:53:52] This error is very annoying 05:40:19 ['/usr/bin/scap', 'deploy-local', '-v', '--repo', 'ores/deploy', '-g', 'worker', 'fetch', '--refresh-config'] on deployment-ores01.deployment-prep.eqiad.wmflabs returned [255]: Permission denied (publickey). [05:57:33] I'm making some progress :D [06:01:43] PROBLEM - Mediawiki Error Rate on graphite-labs is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [10.0] [06:13:14] 10Release-Engineering-Team (Watching / External), 10Wikidata, 10Patch-For-Review, 10User-Addshore: Undeploy the Wikidata extension - https://phabricator.wikimedia.org/T181708#3885567 (10demon) If we're not loading it in config and we're not branching it, then I'd say so yes [06:15:53] (03Abandoned) 10Chad: Use --jobs in git submodule update jobs [integration/config] - 10https://gerrit.wikimedia.org/r/391712 (owner: 10Chad) [06:39:14] This error is now weird, debug1: Offering RSA public key: /etc/keyholder.d/deploy_service.pub [06:39:20] and scap fails [06:39:38] how I can tell a new instance that accepts keyholder keys [06:39:53] I can't find any documentation about that [06:56:07] RECOVERY - Free space - all mounts on deployment-fluorine02 is OK: OK: All targets OK [06:58:03] PROBLEM - Puppet errors on deployment-kafka03 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [07:06:57] RECOVERY - Puppet staleness on deployment-kafka03 is OK: OK: Less than 1.00% above the threshold [3600.0] [07:43:58] 10Release-Engineering-Team, 10Wikidata, 10Epic, 10Patch-For-Review, 10User-Addshore: [Epic] Kill the Wikidata build step - https://phabricator.wikimedia.org/T173818#3885623 (10Addshore) [07:44:02] 10Release-Engineering-Team (Watching / External), 10Wikidata, 10Patch-For-Review, 10User-Addshore: Undeploy the Wikidata extension - https://phabricator.wikimedia.org/T181708#3885622 (10Addshore) 05Open>03Resolved [08:02:44] PROBLEM - Puppet errors on deployment-ores01 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [08:22:39] RECOVERY - Puppet errors on deployment-ores01 is OK: OK: Less than 1.00% above the threshold [0.0] [08:42:56] !log deleted deployment-ores-redis-01 in favor of deployment-ores01 (T184282) [08:43:02] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [08:43:02] T184282: Move beta cluster ORES to its own machine - https://phabricator.wikimedia.org/T184282 [08:43:50] !log changed DNS that ores-beta.wmflabs.org points to deployment-ores01 instead of deployment-sca03 [08:43:55] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [08:45:32] PROBLEM - Host deployment-ores-redis-01 is DOWN: CRITICAL - Host Unreachable (10.68.22.248) [08:47:46] How I can remove it from list of known hosts of icinga? [08:48:12] !log stopping ores services in deployment-sca03 (T184282) [08:48:18] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [08:48:18] T184282: Move beta cluster ORES to its own machine - https://phabricator.wikimedia.org/T184282 [08:48:46] Running puppet agent to make sure any roles is not enabled to bring it back online [08:49:55] !log ladsgroup@deployment-sca03:/srv/deployment$ sudo rm -rf ores (T184282) [08:50:01] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [08:51:51] !log ladsgroup@deployment-tin:~$ mwscript extensions/ORES/maintenance/PopulateDatabase.php --wiki=enwiki (T184276) [08:51:57] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [08:51:57] T184276: Beta Cluster ORES celery worker dies - https://phabricator.wikimedia.org/T184276 [09:26:29] 10Beta-Cluster-Infrastructure, 10Puppet, 10Services (done): Puppet disabled for a month on deployment-restbase0[12] instances - https://phabricator.wikimedia.org/T184477#3885729 (10mobrovac) a:05Pchelolo>03mobrovac That was supposed to be a temporary measure to test reverting a problematic commit. I enab... [09:26:34] 10Beta-Cluster-Infrastructure, 10Puppet, 10Tracking: Deployment-prep hosts with puppet errors (tracking) - https://phabricator.wikimedia.org/T132259#3885733 (10mobrovac) [09:26:36] 10Beta-Cluster-Infrastructure, 10Puppet, 10Services (done): Puppet disabled for a month on deployment-restbase0[12] instances - https://phabricator.wikimedia.org/T184477#3885732 (10mobrovac) 05Open>03Resolved [09:29:54] RECOVERY - Puppet staleness on deployment-restbase01 is OK: OK: Less than 1.00% above the threshold [3600.0] [09:31:25] 10Continuous-Integration-Infrastructure (shipyard), 10Release-Engineering-Team (Kanban), 10Operations: npm 1.4.21 can't use a http proxy - https://phabricator.wikimedia.org/T183569#3885738 (10hashar) [09:35:25] RECOVERY - Puppet staleness on deployment-restbase02 is OK: OK: Less than 1.00% above the threshold [3600.0] [09:37:32] 10Beta-Cluster-Infrastructure, 10ORES, 10Scoring-platform-team (Current), 10User-Ladsgroup, 10Wikimedia-log-errors: Beta Cluster ORES celery worker dies - https://phabricator.wikimedia.org/T184276#3885741 (10Ladsgroup) a:03Ladsgroup [09:38:24] 10Beta-Cluster-Infrastructure, 10ORES, 10Puppet, 10Scoring-platform-team (Current), 10User-Ladsgroup: Puppet broken on deployment-ores01 due to missing hieradata - https://phabricator.wikimedia.org/T184478#3885745 (10Ladsgroup) a:03Ladsgroup [09:42:32] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team, 10Analytics, 10EventBus, and 3 others: What to do with deployment-sca03? - https://phabricator.wikimedia.org/T184501#3885749 (10Ladsgroup) [09:46:18] PROBLEM - Puppet errors on deployment-cassandra3-02 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [10:10:13] PROBLEM - Puppet errors on deployment-cassandra3-01 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [10:21:19] RECOVERY - Puppet errors on deployment-cassandra3-02 is OK: OK: Less than 1.00% above the threshold [0.0] [10:45:11] RECOVERY - Puppet errors on deployment-cassandra3-01 is OK: OK: Less than 1.00% above the threshold [0.0] [10:50:50] 10Continuous-Integration-Infrastructure (shipyard), 10Release-Engineering-Team (Kanban), 10Operations: npm 1.4.21 can't use a http proxy - https://phabricator.wikimedia.org/T183569#3885794 (10hashar) a:03hashar Came back to this the tldr is we have to upgrade to 0.4.3 The Debian bug report is https://bugs... [10:53:58] !log gerrit: created operations/debs/node-tunnel-agent a fork of git://anonscm.debian.org/collab-maint/node-tunnel-agent.git | T183569 [10:54:06] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [10:54:06] T183569: npm 1.4.21 can't use a http proxy - https://phabricator.wikimedia.org/T183569 [10:55:41] (03CR) 10WMDE-leszek: "recheck" [integration/config] - 10https://gerrit.wikimedia.org/r/402862 (https://phabricator.wikimedia.org/T178083) (owner: 10WMDE-leszek) [11:20:37] 10Continuous-Integration-Infrastructure (shipyard), 10Release-Engineering-Team (Kanban), 10Operations, 10Patch-For-Review: npm 1.4.21 can't use a http proxy - https://phabricator.wikimedia.org/T183569#3885851 (10hashar) p:05Triage>03Normal I have forked the Debian repository and bumped the package to 0... [11:22:21] 10Continuous-Integration-Infrastructure (shipyard), 10Release-Engineering-Team (Kanban), 10Kubernetes: Draft requirements for a Kubernetes based solution for CI - https://phabricator.wikimedia.org/T183513#3885858 (10hashar) Per #releng team meeting 01/08: I haven't worked on it, I will do my best to have a f... [12:39:59] (03PS1) 10Chad: Drop export_ext() as dumb wrapper around patching [tools/release] - 10https://gerrit.wikimedia.org/r/403141 [12:59:48] (03PS1) 10Chad: releases: server path code duplication [tools/release] - 10https://gerrit.wikimedia.org/r/403142 [13:08:22] (03CR) 10Addshore: [C: 032] Use npm browser job for data-values/value-view [integration/config] - 10https://gerrit.wikimedia.org/r/402862 (https://phabricator.wikimedia.org/T178083) (owner: 10WMDE-leszek) [13:16:29] (03Merged) 10jenkins-bot: Use npm browser job for data-values/value-view [integration/config] - 10https://gerrit.wikimedia.org/r/402862 (https://phabricator.wikimedia.org/T178083) (owner: 10WMDE-leszek) [13:17:03] dsa ssh host key is no longer supported in gerrit https://gerrit-review.googlesource.com/#/c/gerrit/+/151132/ :) [13:20:13] !log reloaded zuul to deploy https://gerrit.wikimedia.org/r/402862 [13:20:16] (03PS1) 10Chad: make-wmf-branch: Combine skins/vendor to extension logic [tools/release] - 10https://gerrit.wikimedia.org/r/403148 [13:20:18] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [13:24:19] (03CR) 10jerkins-bot: [V: 04-1] make-wmf-branch: Combine skins/vendor to extension logic [tools/release] - 10https://gerrit.wikimedia.org/r/403148 (owner: 10Chad) [13:45:39] (03PS1) 10Hashar: WMF: pin requirements to what is deployed [integration/zuul] (patch-queue/debian/jessie-wikimedia) - 10https://gerrit.wikimedia.org/r/403153 (https://phabricator.wikimedia.org/T158243) [13:49:14] 10Release-Engineering-Team (Watching / External), 10MediaWiki-Platform-Team (MWPT-Q2-Oct-Dec-2017), 10Patch-For-Review, 10Performance-Team (Radar): Support multi-instance hosts on mediawiki-config - https://phabricator.wikimedia.org/T178553#3886371 (10CCicalese_WMF) [13:50:40] (03PS2) 10Chad: make-wmf-branch: Combine skins/vendor to extension logic [tools/release] - 10https://gerrit.wikimedia.org/r/403148 [13:52:11] gerrit's getting securer with ssh https://gerrit-review.googlesource.com/#/c/gerrit/+/151430/ :). [13:53:17] (03CR) 10Chad: [C: 032] Drop export_ext() as dumb wrapper around patching [tools/release] - 10https://gerrit.wikimedia.org/r/403141 (owner: 10Chad) [13:53:52] (03Merged) 10jenkins-bot: Drop export_ext() as dumb wrapper around patching [tools/release] - 10https://gerrit.wikimedia.org/r/403141 (owner: 10Chad) [13:56:28] (03PS1) 10Chad: make-wmf-branch: Remove patching support [tools/release] - 10https://gerrit.wikimedia.org/r/403154 [13:56:34] (03CR) 10Chad: [C: 032] releases: server path code duplication [tools/release] - 10https://gerrit.wikimedia.org/r/403142 (owner: 10Chad) [13:56:39] (03PS2) 10Chad: releases: server path code duplication [tools/release] - 10https://gerrit.wikimedia.org/r/403142 [14:04:44] (03PS1) 10Chad: Fix branch.py [tools/release] - 10https://gerrit.wikimedia.org/r/403155 [14:05:11] (03PS1) 10Hashar: debian: ignore some build artifacts from source [integration/zuul] (debian/jessie-wikimedia) - 10https://gerrit.wikimedia.org/r/403156 [14:05:13] (03PS1) 10Hashar: Refresh patches for Jessie [integration/zuul] (debian/jessie-wikimedia) - 10https://gerrit.wikimedia.org/r/403157 (https://phabricator.wikimedia.org/T158243) [14:15:10] (03CR) 10Chad: [C: 032] Fix branch.py [tools/release] - 10https://gerrit.wikimedia.org/r/403155 (owner: 10Chad) [14:15:43] (03Merged) 10jenkins-bot: Fix branch.py [tools/release] - 10https://gerrit.wikimedia.org/r/403155 (owner: 10Chad) [14:18:27] (03CR) 10Hashar: [C: 032] debian: ignore some build artifacts from source [integration/zuul] (debian/jessie-wikimedia) - 10https://gerrit.wikimedia.org/r/403156 (owner: 10Hashar) [14:18:41] (03PS2) 10Hashar: Refresh patches for Jessie [integration/zuul] (debian/jessie-wikimedia) - 10https://gerrit.wikimedia.org/r/403157 (https://phabricator.wikimedia.org/T158243) [14:20:36] (03PS3) 10Chad: make-wmf-branch: Combine skins/vendor to extension logic [tools/release] - 10https://gerrit.wikimedia.org/r/403148 [14:35:57] (03PS1) 10Hashar: Fix a few lintian issues [integration/zuul] (debian/jessie-wikimedia) - 10https://gerrit.wikimedia.org/r/403160 [14:46:54] (03PS1) 10Chad: branch.py: Simplify [tools/release] - 10https://gerrit.wikimedia.org/r/403162 [14:47:07] (03CR) 10Chad: [C: 032] branch.py: Simplify [tools/release] - 10https://gerrit.wikimedia.org/r/403162 (owner: 10Chad) [14:47:12] (03CR) 10Paladox: [C: 031] "Makes the tests green. Thanks :) :)" [integration/zuul] (debian/jessie-wikimedia) - 10https://gerrit.wikimedia.org/r/403160 (owner: 10Hashar) [14:49:15] (03Merged) 10jenkins-bot: branch.py: Simplify [tools/release] - 10https://gerrit.wikimedia.org/r/403162 (owner: 10Chad) [14:52:23] (03PS2) 10Hashar: Fix a few lintian issues [integration/zuul] (debian/jessie-wikimedia) - 10https://gerrit.wikimedia.org/r/403160 [14:52:25] (03PS3) 10Hashar: Refresh patches for Jessie [integration/zuul] (debian/jessie-wikimedia) - 10https://gerrit.wikimedia.org/r/403157 (https://phabricator.wikimedia.org/T158243) [14:52:35] (03PS1) 10Chad: branch.py: basically make-wmf-branch now [tools/release] - 10https://gerrit.wikimedia.org/r/403165 [14:53:54] !log Change integration/zuul.git HEAD from 'master' to 'patch-queue/debian/jessie-wikimedia' | T158243 T162191 [14:54:01] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [14:54:01] T158243: Update zuul to upstream master - https://phabricator.wikimedia.org/T158243 [14:54:01] T162191: test-requirements.txt in ci-config still points to precise deb - https://phabricator.wikimedia.org/T162191 [14:55:54] 10Continuous-Integration-Config, 10Technical-Debt, 10Zuul: test-requirements.txt in ci-config still points to precise deb - https://phabricator.wikimedia.org/T162191#3886669 (10hashar) In Gerrit I have changed HEAD to point to `patch-queue/debian/jessie-wikimedia` which is the upstream version + our cherry p... [14:56:40] (03PS2) 10Hashar: docker image for commited node_modules [integration/config] - 10https://gerrit.wikimedia.org/r/402876 [14:58:53] (03Draft1) 10Paladox: test do not merge [integration/config] - 10https://gerrit.wikimedia.org/r/403167 [14:58:55] (03PS2) 10Paladox: Switch from precise to jessie in test-requirements.txt [integration/config] - 10https://gerrit.wikimedia.org/r/403167 (https://phabricator.wikimedia.org/T162191) [14:59:29] (03PS1) 10Hashar: debian-glue voting for zuul debian/* branches [integration/config] - 10https://gerrit.wikimedia.org/r/403168 (https://phabricator.wikimedia.org/T158243) [15:02:18] (03CR) 10Hashar: "recheck" [integration/zuul] (debian/jessie-wikimedia) - 10https://gerrit.wikimedia.org/r/403160 (owner: 10Hashar) [15:02:35] (03CR) 10Hashar: [C: 032] debian-glue voting for zuul debian/* branches [integration/config] - 10https://gerrit.wikimedia.org/r/403168 (https://phabricator.wikimedia.org/T158243) (owner: 10Hashar) [15:03:45] (03Merged) 10jenkins-bot: debian-glue voting for zuul debian/* branches [integration/config] - 10https://gerrit.wikimedia.org/r/403168 (https://phabricator.wikimedia.org/T158243) (owner: 10Hashar) [15:04:52] (03CR) 10Hashar: [C: 032] Fix a few lintian issues [integration/zuul] (debian/jessie-wikimedia) - 10https://gerrit.wikimedia.org/r/403160 (owner: 10Hashar) [15:08:33] (03PS1) 10Hashar: Add debian-glue to Zuul gate-and-submit [integration/config] - 10https://gerrit.wikimedia.org/r/403172 (https://phabricator.wikimedia.org/T158243) [15:09:02] (03CR) 10Hashar: [C: 032] Add debian-glue to Zuul gate-and-submit [integration/config] - 10https://gerrit.wikimedia.org/r/403172 (https://phabricator.wikimedia.org/T158243) (owner: 10Hashar) [15:09:43] (03PS4) 10Hashar: Refresh patches for Jessie [integration/zuul] (debian/jessie-wikimedia) - 10https://gerrit.wikimedia.org/r/403157 (https://phabricator.wikimedia.org/T158243) [15:10:06] (03Merged) 10jenkins-bot: Add debian-glue to Zuul gate-and-submit [integration/config] - 10https://gerrit.wikimedia.org/r/403172 (https://phabricator.wikimedia.org/T158243) (owner: 10Hashar) [15:11:54] (03PS1) 10Chad: branch.py: First half of sub-submodule support [tools/release] - 10https://gerrit.wikimedia.org/r/403173 [15:14:23] (03PS5) 10Hashar: Refresh patches for Jessie [integration/zuul] (debian/jessie-wikimedia) - 10https://gerrit.wikimedia.org/r/403157 (https://phabricator.wikimedia.org/T158243) [15:27:43] PROBLEM - Puppet errors on deployment-redis01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [15:33:42] PROBLEM - Puppet errors on deployment-redis02 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [15:53:32] PROBLEM - Free space - all mounts on deployment-kafka01 is CRITICAL: CRITICAL: deployment-prep.deployment-kafka01.diskspace.root.byte_percentfree (<11.11%) [15:59:44] special guest appearance for the second night running! no pictures, please. [16:04:15] 10Gerrit, 10Release-Engineering-Team (Kanban), 10Patch-For-Review, 10Zuul: Update zuul to upstream master - https://phabricator.wikimedia.org/T158243#3886849 (10hashar) Got something build which I am confident to try deploying https://people.wikimedia.org/~hashar/debs/zuul_2.5.0-8-gcbc7f62-wmf6/ Will give... [16:18:30] PROBLEM - Free space - all mounts on deployment-kafka01 is CRITICAL: CRITICAL: deployment-prep.deployment-kafka01.diskspace.root.byte_percentfree (<55.56%) [16:19:58] apergos: on that bzip2 dump thing -- can you check from 'php7.0 -i | grep bzip2' if it thinks the bzip2 streams are enabled in the first place? might also be worth checking if a stray php.ini didn't show up somewhere [16:20:11] (03CR) 10Chad: [C: 032] branch.py: basically make-wmf-branch now [tools/release] - 10https://gerrit.wikimedia.org/r/403165 (owner: 10Chad) [16:20:45] (03Merged) 10jenkins-bot: branch.py: basically make-wmf-branch now [tools/release] - 10https://gerrit.wikimedia.org/r/403165 (owner: 10Chad) [16:21:16] (03CR) 10Chad: [C: 032] branch.py: First half of sub-submodule support [tools/release] - 10https://gerrit.wikimedia.org/r/403173 (owner: 10Chad) [16:21:51] (03Merged) 10jenkins-bot: branch.py: First half of sub-submodule support [tools/release] - 10https://gerrit.wikimedia.org/r/403173 (owner: 10Chad) [16:22:02] brion, sure thing [16:22:22] dumpsgen@deployment-snapshot01:/srv/deployment/dumps/dumps/xmldumps-backup$ php7.0 -i | grep bzip2 [16:22:22] bzip2 compression => disabled (install pecl/bz2) [16:22:29] funky [16:22:30] dum dum DUMM [16:22:42] did that fall out of default install in the 7.x era? [16:23:27] I don't see how [16:24:47] might be a separate php7.0-bz2 package looks like [16:24:50] I mean, [16:25:00] I ran these same dumps a few days ago [16:25:04] heh [16:25:08] was it on 7.0 then too? [16:25:27] 20180107 [16:25:28] oh yes [16:25:34] that's just damn weird :D [16:25:35] and they ran fine [16:25:46] I just grepped for the error in all the syslogs [16:25:52] it's only there for today [16:26:15] that is what's killing me, how did it suddenly stop working, and the only think I can guess is that merge [16:26:18] *thing [16:27:18] wanna revert it and confirm? [16:27:42] it.... shouldn't .... make a difference but spooky action at a distance seems to be the norm now ;) [16:27:52] I wish I could revert it easily by hand [16:28:31] scap doesn't push out a git repo so I can't just cherry-pick a patch in gerrit :-/ [16:29:27] note that on production with php5 [16:30:16] there's a pile of streams listed in response to that command, as we expect... [16:30:23] probably not very enlightening [16:31:51] the other option I have available though is to go ahead and install the extension and see if that "cures" it [16:31:57] because if it does for some godforsaken reason [16:32:03] that saves us a revert right? [16:32:15] apergos: yeah try https://gerrit.wikimedia.org/r/403193 [16:34:05] well I was just going to do it by hand on the instance [16:34:18] if it "works" I can add it to the mediawiki php7 manifest patch right after [16:34:20] that should be fine yeah :D [16:37:10] well. [16:37:13] that works [16:37:25] who the fsck knows why it worked then didn't [16:37:39] * apergos goes to update the patchset [16:38:08] heh [16:38:08] yay [16:38:55] well https://phabricator.wikimedia.org/rGERRITdef5be4c6e7e4cdcbe4300430f635de8c56a52b0 that was quick :). [16:41:57] 10Scap, 10MediaWiki-Platform-Team (MWPT-Q3-Jan-Mar-2018), 10Patch-For-Review: 'sql' script doesn't support multi-instance hosts - https://phabricator.wikimedia.org/T182713#3886951 (10Anomie) 05Open>03Resolved [16:42:57] aaaand cherrypicked onto thedumps puppet master [16:43:15] \o/ [16:44:26] I wonder what other special things we will discover over time [16:44:50] there's a tagline [16:44:56] :D [16:45:01] :-D [16:45:08] Special:Things [16:45:15] lol [16:45:16] run away, run away! [16:45:41] api.php?action=things&thtype=special [16:45:47] yeahno [16:45:52] rofl [16:46:20] ok i better eat breakfast before i gnaw my hand off [16:46:20] tbh I am amazed they run with so little fuss so far [16:46:24] :D [16:46:24] go go go [18:13:01] 10Continuous-Integration-Config, 10Gerrit, 10Cleanup, 10GitHub-Mirrors, and 6 others: Archive mediawiki/extensions/Collection and others - https://phabricator.wikimedia.org/T183891#3887427 (10ovasileva) Currently, the collections extension is still being used to create books and to save them to the books n... [18:32:34] 10Continuous-Integration-Infrastructure, 10MW-1.31-release-notes (WMF-deploy-2018-01-09 (1.31.0-wmf.16)), 10MediaWiki-Platform-Team (MWPT-Q3-Jan-Mar-2018), 10Patch-For-Review: Generate PHP code coverage reports for extensions - https://phabricator.wikimedia.org/T71685#3887470 (10Jdforrester-WMF) [18:33:11] 10Continuous-Integration-Infrastructure, 10MW-1.31-release-notes (WMF-deploy-2018-01-09 (1.31.0-wmf.16)), 10MediaWiki-Platform-Team (MWPT-Q3-Jan-Mar-2018), 10Patch-For-Review: Generate PHP code coverage reports for extensions - https://phabricator.wikimedia.org/T71685#738524 (10Jdforrester-WMF) Hmm. Is thi... [18:38:41] 10Release-Engineering-Team (Kanban), 10User-greg: Basic plan of post-Hackathon team offsite - https://phabricator.wikimedia.org/T184550#3887485 (10greg) p:05Triage>03High [18:39:22] PROBLEM - Puppet errors on deployment-mx is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [18:54:24] (03PS1) 10Chad: branch.py: Cleanup/prep for sub-submodule support [tools/release] - 10https://gerrit.wikimedia.org/r/403219 [18:54:26] (03PS1) 10Chad: grumble grumble submodules [tools/release] - 10https://gerrit.wikimedia.org/r/403220 [18:54:30] PROBLEM - Free space - all mounts on deployment-kafka03 is CRITICAL: CRITICAL: deployment-prep.deployment-kafka03.diskspace.root.byte_percentfree (<100.00%) [18:55:01] (03CR) 10jerkins-bot: [V: 04-1] grumble grumble submodules [tools/release] - 10https://gerrit.wikimedia.org/r/403220 (owner: 10Chad) [18:57:57] PROBLEM - Puppet staleness on deployment-kafka03 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [43200.0] [18:58:43] RECOVERY - Mediawiki Error Rate on graphite-labs is OK: OK: Less than 1.00% above the threshold [1.0] [19:03:54] PROBLEM - puppet last run on contint2001 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [19:09:55] 10Continuous-Integration-Config, 10Proton, 10Readers-Web-Backlog, 10Patch-For-Review: Set up Jenkins for chromium-render and chromium-render-deploy repositories - https://phabricator.wikimedia.org/T179552#3887600 (10ovasileva) [19:12:28] 10Continuous-Integration-Config, 10Proton, 10Readers-Web-Backlog, 10Patch-For-Review: Set up Jenkins for chromium-render and chromium-render-deploy repositories - https://phabricator.wikimedia.org/T179552#3887609 (10ovasileva) p:05Triage>03Normal [19:18:02] i'd still like to abandon https://gerrit.wikimedia.org/r/399248 [19:18:07] just to not see it anymore [19:18:20] it's unknown why i dont have permissions or that happened :) [19:18:34] but i cant get it out of my incoming queue [19:23:49] checks contint2001 puppet [19:25:18] false alert [19:28:54] RECOVERY - puppet last run on contint2001 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:51:36] 10Continuous-Integration-Infrastructure, 10VisualEditor: Provide the same version of Chrome/Chromium in Docker and Nodepool (or migrate qunit for extensions to Docker) - https://phabricator.wikimedia.org/T184102#3887757 (10Deskana) p:05Triage>03Normal [19:55:17] (03PS10) 10Paladox: Backport fix "Fix change number extraction on new enough Gerrit master" [integration/zuul] (debian/jessie-wikimedia) - 10https://gerrit.wikimedia.org/r/356181 (https://phabricator.wikimedia.org/T158243) [19:57:45] (03PS6) 10Paladox: Refresh patches for Jessie [integration/zuul] (debian/jessie-wikimedia) - 10https://gerrit.wikimedia.org/r/403157 (https://phabricator.wikimedia.org/T158243) (owner: 10Hashar) [19:57:47] (03PS11) 10Paladox: [DNM] Backport fix "Fix change number extraction on new enough Gerrit master" [integration/zuul] (debian/jessie-wikimedia) - 10https://gerrit.wikimedia.org/r/356181 (https://phabricator.wikimedia.org/T158243) [19:58:30] (03CR) 10Paladox: "Im not sure why it updated your change but i just wanted to put my test change behind yours so i can test the changes to zuul packaging yo" [integration/zuul] (debian/jessie-wikimedia) - 10https://gerrit.wikimedia.org/r/403157 (https://phabricator.wikimedia.org/T158243) (owner: 10Hashar) [20:01:09] 10Gerrit, 10Release-Engineering-Team, 10releng-201516-q3, 10Developer-Relations, and 3 others: [RfC]: Migrate code review / management from Gerrit to Phabricator - https://phabricator.wikimedia.org/T119908#1866944 (10daniel) Is there still commitment to doing this? [20:03:44] 10Release-Engineering-Team (Kanban), 10Release, 10Train Deployments: 1.31.0-wmf.16 deployment blockers - https://phabricator.wikimedia.org/T180749#3887791 (10mmodell) 05Resolved>03Open a:05demon>03mmodell [20:06:05] 10Gerrit, 10Release-Engineering-Team, 10releng-201516-q3, 10Developer-Relations, and 3 others: [RfC]: Migrate code review / management from Gerrit to Phabricator - https://phabricator.wikimedia.org/T119908#3887802 (10Paladox) @daniel from what i understand it's on hold. [20:06:39] 10Release-Engineering-Team (Kanban), 10Release, 10Train Deployments: 1.31.0-wmf.16 deployment blockers - https://phabricator.wikimedia.org/T180749#3887803 (10mmodell) [20:09:17] 10Gerrit, 10Release-Engineering-Team, 10releng-201516-q3, 10Developer-Relations, and 3 others: [RfC]: Migrate code review / management from Gerrit to Phabricator - https://phabricator.wikimedia.org/T119908#3887810 (10Paladox) p:05Normal>03Lowest Changing to lowest as normal would indicate it will be ha... [20:09:43] PROBLEM - Mediawiki Error Rate on graphite-labs is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [10.0] [20:34:30] PROBLEM - Puppet errors on deployment-eventlogging04 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [20:34:42] RECOVERY - Mediawiki Error Rate on graphite-labs is OK: OK: Less than 1.00% above the threshold [1.0] [20:34:44] 10Gerrit, 10Release-Engineering-Team, 10releng-201516-q3, 10Developer-Relations, and 3 others: [RfC]: Migrate code review / management from Gerrit to Phabricator - https://phabricator.wikimedia.org/T119908#3887916 (10EddieGP) >>! In T119908#3887786, @daniel wrote: > Is there still commitment to doing this?... [20:36:02] 10Gerrit, 10Release-Engineering-Team, 10releng-201516-q3, 10Developer-Relations, and 3 others: [RfC]: Migrate code review / management from Gerrit to Phabricator - https://phabricator.wikimedia.org/T119908#3887931 (10greg) We're (RelEng) going to talk about this at our team offsite (post DevSummit/All hand... [20:36:21] 10Gerrit, 10Release-Engineering-Team, 10releng-201516-q3, 10Developer-Relations, and 3 others: [RfC]: Migrate code review / management from Gerrit to Phabricator - https://phabricator.wikimedia.org/T119908#3887933 (10mmodell) It's essentially deadlocked with indecision. It seems there is no longer any cons... [20:51:36] 10Beta-Cluster-Infrastructure: Request for steward rights on the Beta Cluster - https://phabricator.wikimedia.org/T183790#3887974 (10greg) 05Open>03declined Declining, per reasoning given by @Legoktm [20:57:14] 10Gerrit, 10Release-Engineering-Team (Someday), 10Diffusion, 10User-greg: Diffusion links are broken from within Gerrit - https://phabricator.wikimedia.org/T177277#3887984 (10greg) 05Open>03Resolved a:03greg This work now. [20:57:28] 10Gerrit, 10Release-Engineering-Team (Someday), 10Diffusion, 10User-greg: Kartotherian Diffusion links are broken from within Gerrit - https://phabricator.wikimedia.org/T177277#3887988 (10greg) [21:03:55] 10Beta-Cluster-Infrastructure, 10Operations, 10Puppet: Puppet broken on deployment-netbox, looks like it thinks its a prod box - https://phabricator.wikimedia.org/T184242#3887999 (10greg) (Adding #operations since this box is in support of their evaluation of netbox) [21:04:25] (03CR) 10Hashar: "I guess you somehow rebased the whole chain. This change has not been altered beside you being made the committer https://gerrit.wikimedia" [integration/zuul] (debian/jessie-wikimedia) - 10https://gerrit.wikimedia.org/r/403157 (https://phabricator.wikimedia.org/T158243) (owner: 10Hashar) [21:04:50] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team (Kanban), 10Scap: scap package installed by CI breaks apt and thus puppet - https://phabricator.wikimedia.org/T184118#3888002 (10greg) p:05Triage>03High This is border line UBN!, but it'd be UBN! only if it keeps spitting out non-working scap pack... [21:06:21] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team (Kanban), 10Scap: scap package installed by CI breaks apt and thus puppet - https://phabricator.wikimedia.org/T184118#3888009 (10mmodell) @greg: this isn't an issue currently, however, it does need to be made more robust. [21:12:03] 10Beta-Cluster-Infrastructure, 10Analytics, 10Puppet: Puppet broken on deployment-eventlogging04 due to missing directory '/var/lib/superset'? - https://phabricator.wikimedia.org/T184238#3888024 (10greg) per [[Dev/Maint]] [21:15:16] (03CR) 10Paladox: [C: 031] "Tested on a stretch machine and works." [integration/zuul] (debian/jessie-wikimedia) - 10https://gerrit.wikimedia.org/r/403157 (https://phabricator.wikimedia.org/T158243) (owner: 10Hashar) [21:19:41] greg-g, [[Dev/Maint]] ? [21:20:58] Krenair: my shorthand for https://www.mediawiki.org/wiki/Developers/Maintainers [21:21:21] * greg-g goes to pick up kid, bbiab [21:21:22] greg-g, oh right, yeah [21:21:33] had done assignee but not project. will do in future [21:29:51] (03CR) 10Chad: [C: 032] branch.py: Cleanup/prep for sub-submodule support [tools/release] - 10https://gerrit.wikimedia.org/r/403219 (owner: 10Chad) [21:30:30] (03Merged) 10jenkins-bot: branch.py: Cleanup/prep for sub-submodule support [tools/release] - 10https://gerrit.wikimedia.org/r/403219 (owner: 10Chad) [21:33:10] 10Continuous-Integration-Config, 10Gerrit, 10Cleanup, 10GitHub-Mirrors, and 6 others: Archive mediawiki/extensions/Collection and others - https://phabricator.wikimedia.org/T183891#3888085 (10demon) 05Open>03declined Then we shouldn't do this :) [21:35:55] 10Release-Engineering-Team (Kanban), 10Release, 10Train Deployments: 1.31.0-wmf.16 deployment blockers - https://phabricator.wikimedia.org/T180749#3888110 (10Addshore) [21:47:23] 10Beta-Cluster-Infrastructure, 10Analytics, 10Puppet: Puppet broken on deployment-eventlogging04 due to missing directory '/var/lib/superset'? - https://phabricator.wikimedia.org/T184238#3888157 (10Krenair) I ran the exact same command that puppet does (as the user specified in the puppet file), and it appea... [21:55:53] RECOVERY - Puppet errors on deployment-eventlogging04 is OK: OK: Less than 1.00% above the threshold [0.0] [21:56:41] 10Release-Engineering-Team (Kanban), 10Release, 10Train Deployments: 1.31.0-wmf.16 deployment blockers - https://phabricator.wikimedia.org/T180749#3888233 (10mmodell) [21:56:45] 10Beta-Cluster-Infrastructure, 10Analytics, 10Puppet: Puppet broken on deployment-eventlogging04 due to missing directory '/var/lib/superset'? - https://phabricator.wikimedia.org/T184238#3888234 (10Krenair) 05Open>03Resolved ottomata, -eventlogging04? superset is there? ??... [21:56:48] 10Beta-Cluster-Infrastructure, 10Puppet, 10Tracking: Deployment-prep hosts with puppet errors (tracking) - https://phabricator.wikimedia.org/T132259#3888236 (10Krenair) [22:47:11] (03PS1) 10Hashar: docker: stretch image to run rake [integration/config] - 10https://gerrit.wikimedia.org/r/403311 [22:48:22] (03CR) 10Hashar: "Seems to work with at least mediawiki/ruby/api :]" [integration/config] - 10https://gerrit.wikimedia.org/r/403311 (owner: 10Hashar) [22:49:44] 10Gerrit, 10Upstream: Gerrit should feature customizable message on Login page (No 'Forgot password' link in the gerrit login page.) - https://phabricator.wikimedia.org/T60205#3888411 (10Paladox) Here's the upstream change to support this https://gerrit-review.googlesource.com/#/c/gerrit/+/151192/ [22:57:12] no_justification guessing this is what you want https://gerrit-review.googlesource.com/#/c/gerrit/+/151192/ for ^^? :) [23:26:12] Project mwext-phpunit-coverage-publish build #77: 04FAILURE in 1 hr 42 min: https://integration.wikimedia.org/ci/job/mwext-phpunit-coverage-publish/77/ [23:29:00] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team (Kanban), 10Scap: scap package installed by CI breaks apt and thus puppet - https://phabricator.wikimedia.org/T184118#3888526 (10greg) ack, thanks [23:32:40] 10Continuous-Integration-Config, 10MinusX, 10Google-Code-in-2017, 10MW-1.31-release-notes (WMF-deploy-2018-01-02 (1.31.0-wmf.15)), 10Patch-For-Review: Add MinusX to MediaWiki extensions and PHP library repos - https://phabricator.wikimedia.org/T175794#3888535 (10divadsn) [23:36:09] PROBLEM - Puppet errors on deployment-netbox is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [23:43:15] PROBLEM - Work requests waiting in Zuul Gearman server on contint1001 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [140.0] https://grafana.wikimedia.org/dashboard/db/zuul-gearman?panelId=10fullscreenorgId=1 [23:47:36] jeebus parsoid: https://integration.wikimedia.org/zuul/ [23:49:56] Is there anyway we can get https://gerrit.wikimedia.org/r/#/c/402443/ reviewed and stuff its causing GCI students task to be marked v-1 please :) [23:50:04] https://screenshots.firefox.com/pEhzeoDC1Pq7UtLO/integration.wikimedia.org [23:51:22] Zppix: I would ask the maintainers of that extension to merge it [23:51:44] Ok i thought since it was affecting jenkins its was releng [23:51:48] Sorry [23:51:55] jenkins is failing the test... [23:52:11] oh! [23:52:14] * greg-g misread [23:52:16] sorry [23:52:27] Fatal error: unknown class BsExtensionMW in /home/jenkins/workspace/mwext-testextension-hhvm-composer-jessie/src/extensions/BlueSpiceUserPreferences/UserPreferences.class.php on line 36 [23:52:30] Hehe my fault, my wording is not the best :) [23:53:10] so... how will merging a failing patch fix something?