[01:56:05] PROBLEM - Free space - all mounts on deployment-fluorine02 is CRITICAL: CRITICAL: deployment-prep.deployment-fluorine02.diskspace._srv.byte_percentfree (<11.11%) [01:58:28] 10Project-Admins: Create tags for all Wikimedia projects - https://phabricator.wikimedia.org/T154549 (10martin.monperrus) Done for wikibooks, see https://phabricator.wikimedia.org/T227158 [03:20:56] 10Diffusion, 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO (201907), 10Operations, and 4 others: Cannot connect to vcs@git-ssh.wikimedia.org (since move from phab1001 to phab1003) - https://phabricator.wikimedia.org/T224677 (10Dzahn) The ssh.log is now owned by the $vcs... [05:22:42] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team-TODO (201907), 10Operations, 10serviceops: contint1001 store docker images on separate partition or disk - https://phabricator.wikimedia.org/T207707 (10Joe) @thcipriani why do we even need to save the images? We don't really care about loc... [05:31:00] 10Release-Engineering-Team, 10Release-Engineering-Team-TODO, 10MediaWiki-General-or-Unknown, 10TechCom, and 2 others: Drop PHP 7.0 support from MediaWiki - https://phabricator.wikimedia.org/T216165 (10daniel) This is marked as being blocked on {T225628}. Isn't that the wrong way around? [05:50:23] 10Continuous-Integration-Config, 10BlueSpice: In CI BlueSpice repositories should always have BlueSpiceFoundation injected - https://phabricator.wikimedia.org/T226567 (10Osnard) Is there anything I can help with? [06:21:32] (03PS1) 10Elukey: Archive the puppet zookeeper submodule [integration/config] - 10https://gerrit.wikimedia.org/r/520366 (https://phabricator.wikimedia.org/T227164) [06:41:04] RECOVERY - Free space - all mounts on deployment-fluorine02 is OK: OK: All targets OK [07:04:13] 10Continuous-Integration-Config, 10BlueSpice: In CI BlueSpice repositories should always have BlueSpiceFoundation injected - https://phabricator.wikimedia.org/T226567 (10hashar) For sure! If you feel adventurous you can dig into `zuul/parameter_function.py` and attempt to add the dependency. I haven't even lo... [07:08:07] 10Continuous-Integration-Config, 10BlueSpice: In CI BlueSpice repositories should always have BlueSpiceFoundation injected - https://phabricator.wikimedia.org/T226567 (10hashar) Or better, I have added a feature to #quibble which makes it process the dependencies as they are defined in `extension.json`. So tha... [07:16:15] 10Continuous-Integration-Config, 10BlueSpice: In CI BlueSpice repositories should always have BlueSpiceFoundation injected - https://phabricator.wikimedia.org/T226567 (10Osnard) This would indeed be awesome! Actually most (all?) of our extensions are already making use of the `requires.extensions` field in `ex... [07:42:53] 10Continuous-Integration-Config, 10Patch-For-Review: Run phan secheck on PHP 7.2, not PHP 7.0 - https://phabricator.wikimedia.org/T226420 (10Daimona) [07:42:58] 10Continuous-Integration-Config, 10phan-taint-check-plugin, 10Patch-For-Review: Upgrade php-ast to 1.0.1 in CI containers - https://phabricator.wikimedia.org/T218719 (10Daimona) [07:46:12] 10Continuous-Integration-Config, 10Patch-For-Review: Run phan secheck on PHP 7.2, not PHP 7.0 - https://phabricator.wikimedia.org/T226420 (10Daimona) [07:46:47] 10Release-Engineering-Team-TODO (201907), 10Security-Team, 10phan-taint-check-plugin: Phan-taint-check-plugin not available for PHP > 7.0 - https://phabricator.wikimedia.org/T207344 (10Daimona) [07:47:05] 10Release-Engineering-Team-TODO (201907), 10Security-Team, 10phan-taint-check-plugin: Phan-taint-check-plugin not available for PHP > 7.0 - https://phabricator.wikimedia.org/T207344 (10Daimona) [08:01:24] 10Continuous-Integration-Config, 10LibUp, 10phan-taint-check-plugin: Upgrade taint-check to 2.0 in all repos - https://phabricator.wikimedia.org/T227172 (10Daimona) [08:01:39] 10Continuous-Integration-Config, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO, 10LibUp: LibraryUpgrader CI normalisation tasks, June/July 2019 - https://phabricator.wikimedia.org/T225325 (10Daimona) [08:01:41] 10Continuous-Integration-Config, 10LibUp, 10phan-taint-check-plugin: Upgrade taint-check to 2.0 in all repos - https://phabricator.wikimedia.org/T227172 (10Daimona) [08:02:07] 10Continuous-Integration-Config, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO, 10LibUp: LibraryUpgrader CI normalisation tasks, June/July 2019 - https://phabricator.wikimedia.org/T225325 (10Daimona) [08:02:24] 10Continuous-Integration-Config, 10LibUp, 10phan-taint-check-plugin: Upgrade taint-check to 2.0 in all repos - https://phabricator.wikimedia.org/T227172 (10Daimona) [08:02:51] 10Continuous-Integration-Config, 10Patch-For-Review: Run phan secheck on PHP 7.2, not PHP 7.0 - https://phabricator.wikimedia.org/T226420 (10Daimona) [08:02:53] 10Continuous-Integration-Config, 10LibUp, 10phan-taint-check-plugin: Upgrade taint-check to 2.0 in all repos - https://phabricator.wikimedia.org/T227172 (10Daimona) [08:03:02] 10Continuous-Integration-Config, 10Patch-For-Review: Run phan secheck on PHP 7.2, not PHP 7.0 - https://phabricator.wikimedia.org/T226420 (10Daimona) [08:03:04] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team-TODO (201907), 10Operations, 10serviceops: contint1001 store docker images on separate partition or disk - https://phabricator.wikimedia.org/T207707 (10hashar) >>! In T207707#5302475, @Joe wrote: > @thcipriani why do we even need to save t... [08:05:26] 10Release-Engineering-Team, 10Release-Engineering-Team-TODO, 10Operations, 10SRE-Access-Requests: Request access to deployment cluster for Alaa Sarhan - https://phabricator.wikimedia.org/T223698 (10MoritzMuehlenhoff) @alaa_wmde : Please generate a separate SSH key for the access to the Wikimedia production... [08:07:35] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO: Regularly run mwext-{name}-testextension-* jobs to make sure they are still passing after core or dependency changes - https://phabricator.wikimedia.org/T93414 (10hashar) Zuul should... [08:09:46] 10Continuous-Integration-Infrastructure, 10Operations: Jessie rsyslog_8.1901.0-1~bpo8+wmf1_amd64.deb package fails to upgrade - https://phabricator.wikimedia.org/T222166 (10hashar) That the package is broken? I am stalling this and filling another task for WMCS to rebuild the Jessie image. [08:09:54] 10Continuous-Integration-Infrastructure, 10Operations: Jessie rsyslog_8.1901.0-1~bpo8+wmf1_amd64.deb package fails to upgrade - https://phabricator.wikimedia.org/T222166 (10hashar) 05Open→03Stalled [08:10:30] 10Continuous-Integration-Config: Move all CI generic tasks from PHP70 to PHP72 - https://phabricator.wikimedia.org/T225457 (10Daimona) [08:10:32] 10Continuous-Integration-Config, 10Patch-For-Review: Run phan secheck on PHP 7.2, not PHP 7.0 - https://phabricator.wikimedia.org/T226420 (10Daimona) 05Stalled→03Open Copying what I said in T227172: the current version (1.5.1) requires PHP70 jobs, while the new version (2.0.0) can run on PHP70+. I believe... [08:11:32] (03CR) 10Awight: Support running PHPUnit unit tests (031 comment) [integration/quibble] - 10https://gerrit.wikimedia.org/r/519166 (https://phabricator.wikimedia.org/T87781) (owner: 10Kosta Harlan) [08:12:33] 10Continuous-Integration-Infrastructure, 10Operations: Jessie rsyslog_8.1901.0-1~bpo8+wmf1_amd64.deb package fails to upgrade - https://phabricator.wikimedia.org/T222166 (10MoritzMuehlenhoff) The package per se isn't broken, only the upgrade path from the old jessie version to 8.1901.0-1~bpo8+wmf1, given that... [08:13:10] (03CR) 10Awight: Run node browser tests in each repo (031 comment) [integration/quibble] - 10https://gerrit.wikimedia.org/r/510709 (https://phabricator.wikimedia.org/T199116) (owner: 10Awight) [08:14:06] (03CR) 10Awight: Run node browser tests in each repo (031 comment) [integration/quibble] - 10https://gerrit.wikimedia.org/r/510709 (https://phabricator.wikimedia.org/T199116) (owner: 10Awight) [08:14:18] 10Continuous-Integration-Infrastructure, 10cloud-services-team: Please reubild Jessie base image due to rsyslog upgrade breakage - https://phabricator.wikimedia.org/T227173 (10hashar) [08:14:58] 10Continuous-Integration-Infrastructure, 10Cloud-VPS, 10cloud-services-team: Please reubild Jessie base image due to rsyslog upgrade breakage - https://phabricator.wikimedia.org/T227173 (10hashar) [08:18:21] (03PS1) 10Awight: Use the npm --prefer-offline flag [integration/quibble] - 10https://gerrit.wikimedia.org/r/520380 [08:20:22] 10Continuous-Integration-Infrastructure, 10Operations: Jessie rsyslog_8.1901.0-1~bpo8+wmf1_amd64.deb package fails to upgrade - https://phabricator.wikimedia.org/T222166 (10hashar) >>! In T222166#5302794, @MoritzMuehlenhoff wrote: > The package per se isn't broken, only the upgrade path from the old jessie ver... [08:26:04] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team, 10Release-Engineering-Team-TODO, 10MediaWiki-Configuration: Fatal error Cannot redeclare wmfLabsSettings() on Beta cluster wikis - https://phabricator.wikimedia.org/T224899 (10hashar) 05Resolved→03Open The `require_once` is a workaround but t... [08:32:00] (03CR) 10Hashar: [C: 03+2] Archive the puppet zookeeper submodule [integration/config] - 10https://gerrit.wikimedia.org/r/520366 (https://phabricator.wikimedia.org/T227164) (owner: 10Elukey) [08:34:58] (03Merged) 10jenkins-bot: Archive the puppet zookeeper submodule [integration/config] - 10https://gerrit.wikimedia.org/r/520366 (https://phabricator.wikimedia.org/T227164) (owner: 10Elukey) [08:35:56] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO: Regularly run mwext-{name}-testextension-* jobs to make sure they are still passing after core or dependency changes - https://phabricator.wikimedia.org/T93414 (10Tgr) Maybe start as... [08:38:20] 10Gerrit, 10Security: Cannot assign user name "XXX" to account ####; name already in use. - https://phabricator.wikimedia.org/T216605 (10hashar) Congratulations @thcipriani ! [08:49:32] 10Continuous-Integration-Infrastructure, 10Cloud-VPS, 10cloud-services-team: Please reubild Jessie base image due to rsyslog upgrade breakage - https://phabricator.wikimedia.org/T227173 (10aborrero) Side node: you should try not building anything new with Jessie at this point. [09:07:51] (03PS9) 10Awight: Run node browser tests in each repo [integration/quibble] - 10https://gerrit.wikimedia.org/r/510709 (https://phabricator.wikimedia.org/T199116) [09:07:59] (03CR) 10jerkins-bot: [V: 04-1] Run node browser tests in each repo [integration/quibble] - 10https://gerrit.wikimedia.org/r/510709 (https://phabricator.wikimedia.org/T199116) (owner: 10Awight) [09:08:49] 10Continuous-Integration-Config, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO, 10LibUp: LibraryUpgrader CI normalisation tasks, June/July 2019 - https://phabricator.wikimedia.org/T225325 (10Daimona) [09:08:51] 10Continuous-Integration-Config, 10LibUp, 10phan-taint-check-plugin: Upgrade taint-check to 2.0 in all repos - https://phabricator.wikimedia.org/T227172 (10Daimona) 05Open→03Stalled [09:11:15] (03PS10) 10Awight: Run node browser tests in each repo [integration/quibble] - 10https://gerrit.wikimedia.org/r/510709 (https://phabricator.wikimedia.org/T199116) [09:22:31] 10Gerrit: Enable sandbox branches in gerrit - https://phabricator.wikimedia.org/T227159 (10Aklapper) This does not seem to be about #gerrit-privilege-requests but about #Gerrit [09:33:18] 10Continuous-Integration-Config, 10BlueSpice: In CI BlueSpice repositories should always have BlueSpiceFoundation injected - https://phabricator.wikimedia.org/T226567 (10hashar) The feature merely inspect `extension.json` and then recursively clone the required repositories, fetch the proper patch/branch. Once... [09:34:11] 10Continuous-Integration-Infrastructure, 10Cloud-VPS, 10cloud-services-team: Please rebuild Jessie base image due to rsyslog upgrade breakage - https://phabricator.wikimedia.org/T227173 (10hashar) [09:34:55] 10Continuous-Integration-Infrastructure, 10Cloud-VPS, 10cloud-services-team: Please rebuild Jessie base image due to rsyslog upgrade breakage - https://phabricator.wikimedia.org/T227173 (10hashar) >>! In T227173#5302869, @aborrero wrote: > Side node: you should try not building anything new with Jessie at th... [09:42:55] 10Gerrit: Enable sandbox branches in gerrit - https://phabricator.wikimedia.org/T227159 (10hashar) 05Open→03Resolved a:03hashar User sandboxes are enabled for all projects. From `All-Projects.git` in `refs/meta/config`: ` name=project.config [access "refs/heads/sandbox/${username}/*"] create = group Re... [09:48:50] 10Release-Engineering-Team-TODO (201907), 10MediaWiki-Core-Testing, 10Quibble, 10MW-1.34-notes (1.34.0-wmf.7; 2019-05-28), and 2 others: Quibble should run `npm install` and `npm run selenium-test` for each extension/skin that has Selenium tests - https://phabricator.wikimedia.org/T199116 (10awight) I've a... [10:52:46] 10Project-Admins, 10MediaWiki-Documentation, 10Documentation, 10Proposal: Consider archiving #MediaWiki-Documentation in favour of single #Documentation - https://phabricator.wikimedia.org/T142345 (10Aklapper) It feels to me like this is blocked on T76942... [10:56:57] 10Gerrit, 10Technical-Tool-Request, 10Code-Health: Set up gerritstats on Toolforge - https://phabricator.wikimedia.org/T215735 (10Aklapper) Adding #Code-health because some features might help with identifying review performance [11:55:14] 10Release-Engineering-Team, 10Operations, 10SRE-Access-Requests: Request access to deployment cluster for Jakob_WMDE - https://phabricator.wikimedia.org/T227193 (10WMDE-leszek) [11:56:06] 10Release-Engineering-Team, 10Operations, 10SRE-Access-Requests: Request access to deployment cluster for Jakob_WMDE - https://phabricator.wikimedia.org/T227193 (10WMDE-leszek) I am an engineering manager at WMDE, and Jakob's line manager. By submitting this request I approve it at WMDE's end. [12:05:51] 10Release-Engineering-Team, 10Operations, 10SRE-Access-Requests: Request access to deployment cluster for Jakob_WMDE - https://phabricator.wikimedia.org/T227193 (10MoritzMuehlenhoff) Adding @greg for approval. [12:23:51] 10Continuous-Integration-Config, 10MediaWiki-Core-Testing, 10MW-1.31-release, 10MW-1.32-release, 10phan: Decide on future of running Phan tests on release branches - https://phabricator.wikimedia.org/T226945 (10Reedy) >>! In T226945#5302166, @Jdforrester-WMF wrote: > Yeah, I'd propose just running them o... [12:30:18] (03PS3) 10Daimona Eaytoy: Add tests for PerfCheck [integration/config] - 10https://gerrit.wikimedia.org/r/517447 [12:48:02] 10Release-Engineering-Team, 10Operations, 10SRE-Access-Requests: Request access to deployment cluster for Jakob_WMDE - https://phabricator.wikimedia.org/T227193 (10MoritzMuehlenhoff) p:05Triage→03Normal [12:55:02] (03PS6) 10Giuseppe Lavagetto: Remove functionality to talk to conftool [tools/scap] - 10https://gerrit.wikimedia.org/r/491412 [12:57:38] (03CR) 10Volans: "I'm not familiar with this code base, but LGTM" [tools/scap] - 10https://gerrit.wikimedia.org/r/491412 (owner: 10Giuseppe Lavagetto) [12:57:41] <_joe_> thcipriani: I'm going to be bold and merge ^^, as conftool is now python3-only and this is blocking deployment of new versions [12:58:32] _joe_: thumbs-up [12:58:35] <_joe_> I'll wait for your ack before packaging a new version / deploying [12:58:36] <_joe_> lol [12:58:39] <_joe_> you're already here? [12:58:59] <_joe_> or do you have a fitbit that electrocutes you whenever I ping you? :P [12:59:17] pretty much the latter [12:59:45] on my phone. I'll check it out when I'm back at my desk. [13:00:54] (03CR) 10Giuseppe Lavagetto: [C: 03+2] Remove functionality to talk to conftool [tools/scap] - 10https://gerrit.wikimedia.org/r/491412 (owner: 10Giuseppe Lavagetto) [13:03:39] (03Merged) 10jenkins-bot: Remove functionality to talk to conftool [tools/scap] - 10https://gerrit.wikimedia.org/r/491412 (owner: 10Giuseppe Lavagetto) [13:04:30] (03CR) 10jenkins-bot: Remove functionality to talk to conftool [tools/scap] - 10https://gerrit.wikimedia.org/r/491412 (owner: 10Giuseppe Lavagetto) [13:09:43] awight, hi :) [13:09:53] o/. This job should have pulled the new config, https://integration.wikimedia.org/ci/job/beta-mediawiki-config-update-eqiad/lastBuild/console [13:10:05] HEAD is now at 237b65b0e Enable experimental FileImporter features on labs [13:10:57] awight, it's on deployment-deploy01 at least [13:11:20] I just ran "scap pull" for fun... [13:11:27] I've ran scap sync-file for more fun :D [13:11:30] Now it's there [13:12:35] Well. That solves our problem for now, thanks for the sanity check! [13:12:37] !log deployment-prep Manually ran sudo -u jenkins-deploy scap sync-file wmf-config/CommonSettings-labs.php 'Not synced yet?' to sync file that's mysteriously not synced, but present on deployment host [13:12:38] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [13:13:24] awight, yw. Hoping your code works :) [13:14:39] awight, not sure if this shouldn't be reported somehow, what do you think? [13:14:44] (this==half-deployed code) [13:16:41] awight, btw, https://phabricator.wikimedia.org/P8707 is output of my sync-file attempt [13:16:51] interesting thing is "failed to update opcache" [13:18:03] The original failure, I can't say much about. maybe what happened is that your sync-file ran at the same time as the automatic beta config sync job? [13:18:34] awight, that happened before, and scap complained "someone else is running me" [13:18:35] I'm willing to ignore this one but proceed with elevated suspicion for the next time :-) [13:18:54] haha humor setting 60% [13:19:15] as you think :) [13:19:34] (03PS1) 10Ppchelko: Build and publish RESTBase image on branch push. [integration/config] - 10https://gerrit.wikimedia.org/r/520437 (https://phabricator.wikimedia.org/T226536) [13:19:35] I thought a scap wouldn't be necessary for labs config? [13:19:55] it shouldn't be [13:20:01] that magic should happen automatically [13:21:35] * Urbanecm is looking into beta logstash [13:21:37] (03CR) 10jerkins-bot: [V: 04-1] Build and publish RESTBase image on branch push. [integration/config] - 10https://gerrit.wikimedia.org/r/520437 (https://phabricator.wikimedia.org/T226536) (owner: 10Ppchelko) [13:26:21] 10Continuous-Integration-Config, 10Release Pipeline, 10serviceops-radar, 10Core Platform Team (RESTBase Split (CDP2)), and 3 others: Trigger RESTRouter image builds on push/tag - https://phabricator.wikimedia.org/T226536 (10Pchelolo) After discussion with @mobrovac we have decided to go with publishing on... [13:43:45] (03CR) 10Kosta Harlan: Support running PHPUnit unit tests (031 comment) [integration/quibble] - 10https://gerrit.wikimedia.org/r/519166 (https://phabricator.wikimedia.org/T87781) (owner: 10Kosta Harlan) [13:48:02] (03CR) 10Awight: Support running PHPUnit unit tests (031 comment) [integration/quibble] - 10https://gerrit.wikimedia.org/r/519166 (https://phabricator.wikimedia.org/T87781) (owner: 10Kosta Harlan) [13:49:54] (03CR) 10Kosta Harlan: Support running PHPUnit unit tests (031 comment) [integration/quibble] - 10https://gerrit.wikimedia.org/r/519166 (https://phabricator.wikimedia.org/T87781) (owner: 10Kosta Harlan) [13:54:36] (03CR) 10Awight: Support running PHPUnit unit tests (031 comment) [integration/quibble] - 10https://gerrit.wikimedia.org/r/519166 (https://phabricator.wikimedia.org/T87781) (owner: 10Kosta Harlan) [14:24:38] (03PS2) 10Ppchelko: Build and publish RESTBase image on branch push. [integration/config] - 10https://gerrit.wikimedia.org/r/520437 (https://phabricator.wikimedia.org/T226536) [14:26:10] (03CR) 10jerkins-bot: [V: 04-1] Build and publish RESTBase image on branch push. [integration/config] - 10https://gerrit.wikimedia.org/r/520437 (https://phabricator.wikimedia.org/T226536) (owner: 10Ppchelko) [14:34:08] hi, where can I verify that this job https://integration.wikimedia.org/ci/view/Selenium/job/selenium-daily-beta-RelatedArticles/332/ is properly sending emails on failures and/or IRC notifications? (checked my IRC logs but could not find this particular failure) [14:37:19] (03PS3) 10Ppchelko: Build and publish RESTBase image on branch push. [integration/config] - 10https://gerrit.wikimedia.org/r/520437 (https://phabricator.wikimedia.org/T226536) [14:39:43] found https://lists.wikimedia.org/pipermail/qa-alerts/2019-June/thread.html <= this ML is great [14:41:41] _joe_: FYI, latest scap is live on beta, works fine, change lgtm: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/256331/console [14:42:01] (03CR) 10Ppchelko: "I'm seeking help here. RESTBase is currently still primarily developed in GH, so we don't really need to submit for the review in gerrit, " [integration/config] - 10https://gerrit.wikimedia.org/r/520437 (https://phabricator.wikimedia.org/T226536) (owner: 10Ppchelko) [14:42:02] <_joe_> thcipriani: oh you already built a release? [14:42:26] release for beta gets built automagically, I just did the cumin thing to install. [14:42:53] (built from "master" rather than from "release", that is) [14:43:23] 10Continuous-Integration-Config, 10Release Pipeline, 10serviceops-radar, 10Core Platform Team (RESTBase Split (CDP2)), and 3 others: Trigger RESTRouter image builds on push/tag - https://phabricator.wikimedia.org/T226536 (10Pchelolo) Ok, doing what's proposed in the task description doesn't really work, ca... [14:43:47] <_joe_> thcipriani: ok about that [14:44:15] <_joe_> I've seen lately you've changed version.py and the debian/changelog only in the release branch [14:44:23] <_joe_> should I follow that convention? [14:45:45] that was a mistake this last time [14:45:59] I should add that to the instructions: bump master to a version ahead [14:46:31] so that beta stays up-to-date [14:47:45] icymi I have instructions that I follow when making a new release pre they typical serviceops handoff: https://wikitech.wikimedia.org/wiki/Scap3#Production_Upgrade [14:59:42] (03PS1) 10DCausse: Add discovery alerts ML to some beta selenium jobs [integration/config] - 10https://gerrit.wikimedia.org/r/520460 (https://phabricator.wikimedia.org/T225225) [15:20:52] 10Gerrit, 10Technical-Tool-Request, 10Code-Health: Set up gerritstats on Toolforge - https://phabricator.wikimedia.org/T215735 (10hashar) p:05Triage→03Normal [15:28:44] (03PS4) 10Ppchelko: Build and publish RESTRouter image. [integration/config] - 10https://gerrit.wikimedia.org/r/520437 (https://phabricator.wikimedia.org/T226536) [16:14:26] (03Abandoned) 10Pablo Grass (WMDE): codehealth: send failures via email [integration/config] - 10https://gerrit.wikimedia.org/r/519403 (owner: 10Pablo Grass (WMDE)) [16:27:01] 10Beta-Cluster-Infrastructure, 10CirrusSearch, 10Wikidata Termox, 10Discovery-Search (Current work), and 2 others: PHP Warning: Invalid argument supplied for foreach() - https://phabricator.wikimedia.org/T226969 (10debt) 05Open→03Resolved [16:31:28] 10Gerrit: Enable sandbox branches in gerrit - https://phabricator.wikimedia.org/T227159 (10MarkAHershberger) 05Resolved→03Open I understand the limited usefulness for development. I also understand that I can create these branches and that they are public: ` $ git ls-remote https://gerrit.wikimedia.org/r/me... [16:32:03] 10Gerrit: Enable sandbox branches in gerrit - https://phabricator.wikimedia.org/T227159 (10MarkAHershberger) [16:35:23] Hey releng: for https://wikitech.wikimedia.org/wiki/Deployments, can I just edit https://wikitech.wikimedia.org/wiki/Deployments/Template and swap Brian's info for mine for the security window, for both the Euro and American templates? [16:35:35] sbassett: Yup. [16:35:48] Rad, thanks James_F [16:35:53] Sorry, meant to do that earlier, but if you do it that's even better. :-) [16:38:46] Ok, looks I did get added to the template, just hasn't made it over to /Deployments yet. Though I did remove Brian. No sense in pinging him if he's on irc :) [16:47:10] James_F: got an issue on our hands, the wikidata json entity dumps are broken due to https://gerrit.wikimedia.org/r/#/c/mediawiki/extensions/Wikibase/+/518833/ ... tracked down by hoo (why is ne not in this channel? I shall summon him) [16:48:03] this is week two of them being broken so it's ubn for wikidata folks. how bad is this phan error? [16:48:18] a revert is the clean approach... the hash content is needed in these dumps [16:48:37] apergos: This was the "Wikibase won't run phan on php72 and so the entire repo is broken" fix. [16:49:04] apergos: Also the original was by Michael and merged by Amir1, I just back-ported it so that we could back-port other code. [16:49:24] mind repeating the previous comment for hoo? [16:49:27] now that he's here [16:49:32] Oh, sure. [16:50:02] Hey hoo. https://gerrit.wikimedia.org/r/c/mediawiki/extensions/Wikibase/+/517836 was the fix made by Michael to make Wikibase not fail phan in php72. [16:50:07] we're just looking for the way to unbreak this (and conscious that no deploys tomorrow, or friday, and the run will have already started on monday) [16:50:23] If we revert it, master will be broken. [16:50:42] I could switch phan into non-voting for Wikibase, but all dependent repos would also die. [16:50:53] Given that Wikibase is in the gate, that's all repos. [16:51:16] So reverting it is not really a plan. [16:51:22] this is where i curse the day I decided to become a software developer [16:51:32] Dependencies! Aren't they great? [16:51:39] just grand [16:52:10] Ok, we have two options now a) I'll try to build a proper fix ASAP b) we hardcoded the constant values in there for now [16:52:19] Hard-coding WFM. [16:52:46] And we can back-port. [16:52:56] do what ya gotta do [16:53:14] Exactly. [16:53:29] (Though how were these constants marked as deprecated if they're still needed, etc. etc.) [16:54:17] ("Let's see if tests fail." guess there might not be comprehensive tests for this bit... or people don't run them, or who knows) [16:54:48] thanks for being willing to backport on short notice [16:54:59] Sure, but constants are the one thing that's easy to spot in code search. :-) [16:55:15] oh pshaw, a grep -r? :-P [16:55:34] Yeah yeah, there I go having all these fancy expectations. :-) [16:55:40] :-D [16:55:56] James_F: apergos: https://gerrit.wikimedia.org/r/520496 [16:56:01] Fixes the problem locally [16:56:22] C+2'ed. [16:56:32] Thanks [16:56:51] Something is seriously awry here… I guess this function is not actually used in the way it is described [16:56:57] but that is for sometime else to figure out [16:56:57] :-( [16:57:06] Yeah, UBN fix, then remedy. [16:57:09] and we have no integration tests including qualifiers apparently (wtf) [16:57:10] yep [16:57:22] that seems... problematic [16:57:24] It's not like qualifiers are a big part of Wikibase. ;-) [16:57:40] I imagine the integration tests were written before they were added into the model and never extended? [16:58:07] sounds likely [16:58:17] Possible… or whoever did this (probably me) was to lazy to setup all of this [17:02:08] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team-TODO (201907), 10Operations, 10serviceops: contint1001 store docker images on separate partition or disk - https://phabricator.wikimedia.org/T207707 (10Dzahn) >>! In T207707#5302139, @thcipriani wrote: > The other option would be to move t... [17:02:14] once again zuul-watching for fun and profit... [17:13:35] I figured why out integration tests didn't catch this https://gerrit.wikimedia.org/r/c/mediawiki/extensions/Wikibase/+/520501 [17:13:46] what are the red bits in the zuul progress bar? [17:14:14] ohsigh [17:14:16] Flaky browser tests [17:14:27] welp now we know [17:18:48] 10Gerrit: Enable sandbox branches in gerrit - https://phabricator.wikimedia.org/T227159 (10greg) p:05Triage→03Low To unblock Mark's work... @paladox, could @MarkAHershberger use your WMCS install of gerrit to test his branching code? [17:19:46] 10Gerrit: Enable sandbox branches in gerrit - https://phabricator.wikimedia.org/T227159 (10Paladox) Yup! (which is at https://gerrit.git.wmflabs.org/r/) i can make you an admin once you sign in (also need to register at https://ldapauth-gitldap.wmflabs.org/w/index.php) [17:22:53] Hah, the fixed integration test fails on master :) [17:27:44] ahahaha of course [17:28:04] so in zuul i see this patchset at the end of the queue now (probably for the 'verified' piece)? grrrr [17:41:38] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team-TODO (201907), 10Operations, 10Release Pipeline: Switch CI Docker Storage Driver to its own partition and to use devicemapper - https://phabricator.wikimedia.org/T178663 (10Dzahn) See progress on T207707 . The new disks are mounted now an... [17:44:58] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team-TODO (201907), 10Operations: contint1001: DISK WARNING - free space: /srv 88397 MB (10% inode=94%): - https://phabricator.wikimedia.org/T219850 (10Dzahn) Can be closed together with T207707 once the docker images have moved to the new logica... [17:45:47] oh, it was the stinking browser tests. grrr and now we are at the bottom of the queue after all that [17:58:28] (03CR) 10Jforrester: [C: 03+2] Use the npm --prefer-offline flag [integration/quibble] - 10https://gerrit.wikimedia.org/r/520380 (owner: 10Awight) [17:59:09] (03Merged) 10jenkins-bot: Use the npm --prefer-offline flag [integration/quibble] - 10https://gerrit.wikimedia.org/r/520380 (owner: 10Awight) [17:59:39] (03CR) 10jenkins-bot: Use the npm --prefer-offline flag [integration/quibble] - 10https://gerrit.wikimedia.org/r/520380 (owner: 10Awight) [18:07:06] 10Gerrit: Enable sandbox branches in gerrit - https://phabricator.wikimedia.org/T227159 (10MarkAHershberger) I got `⧼authmanager-authplugin-create-fail⧽` when I tried to create an account. [18:12:58] 10Gerrit: Enable sandbox branches in gerrit - https://phabricator.wikimedia.org/T227159 (10Paladox) @MarkAHershberger try again :) [18:14:13] postmerge... [18:14:38] 10Continuous-Integration-Config, 10Patch-For-Review: Run phan secheck on PHP 7.2, not PHP 7.0 - https://phabricator.wikimedia.org/T226420 (10Jdforrester-WMF) This is directly blocked by T227172 which is itself Stalled, so surely this is also Stalled? [18:16:36] 10Continuous-Integration-Config, 10phan-taint-check-plugin, 10Patch-For-Review: Upgrade php-ast to 1.0.1 in CI containers - https://phabricator.wikimedia.org/T218719 (10Jdforrester-WMF) >>! In T218719#5300491, @Daimona wrote: >>>! In T218719#5300479, @Jdforrester-WMF wrote: >>>>! In T218719#5300256, @Daimona... [18:17:46] hoo: Deploy time? [18:19:08] 10Gerrit: Enable sandbox branches in gerrit - https://phabricator.wikimedia.org/T227159 (10MarkAHershberger) Now, an xdebug dump: ` Notice: Trying to get property 'status' of non-object in /vagrant/mediawiki/includes/auth/AuthManager.php on line 1347 ` and several complaints about headers already being sent (nat... [18:19:39] James_F: Would be nice to go now (and not have to wait for 1am) [18:19:53] hoo: Of course. [18:21:54] I could be here if I had to but [18:21:58] now would be really pleassant [18:22:08] (I'm deploying it now.) [18:22:17] \o/ [18:22:40] apergos: Are there any dump processes we need to kill off or are they all complete (but wrong)? [18:23:02] no, these finished yesterday [18:23:08] * James_F nods. [18:23:25] so tne new ones of this type go next monday at ridiculous-o-clock in the morning my time [18:23:36] Unless we re-trigger? [18:23:44] decision made not to rerun [18:23:50] there are ones in an alternate format [18:23:54] OK. [18:24:06] not so convenient (rdf vs json) but there it is [18:24:13] 10Continuous-Integration-Config, 10Patch-For-Review: Run phan secheck on PHP 7.2, not PHP 7.0 - https://phabricator.wikimedia.org/T226420 (10Daimona) @Jdforrester-WMF Well, T227172 is definitely a blocker and for that part yes, this one is stalled. However, as I said above, the seccheck containers have to be u... [18:24:14] * James_F nods. [18:24:59] 10Continuous-Integration-Config, 10Patch-For-Review: Run phan secheck on PHP 7.2, not PHP 7.0 - https://phabricator.wikimedia.org/T226420 (10Jdforrester-WMF) Anyway, I marked this as Stalled on T218719 in the first place. :-) [18:27:49] * apergos goes back to zuul-watching [18:45:41] verified +2 ready-to-submit [18:46:18] merged! [18:46:51] 10Continuous-Integration-Config, 10Patch-For-Review: Run phan secheck on PHP 7.2, not PHP 7.0 - https://phabricator.wikimedia.org/T226420 (10Daimona) Yeah that makes sense :) TBH I thought that upgrading ast would have been easier, but if there's no clear solution then yes, I guess this can go back stalled. [18:55:20] looks like postmerge finished too [19:00:10] thanks much for helping to make this emergency go away, James F [20:14:58] PROBLEM - Puppet errors on integration-slave-jessie-1004 is CRITICAL: (Service Check Timed Out) [20:15:22] PROBLEM - Puppet errors on integration-slave-jessie-1002 is CRITICAL: (Service Check Timed Out) [20:15:42] PROBLEM - Puppet staleness on deployment-fluorine02 is CRITICAL: (Service Check Timed Out) [20:15:46] PROBLEM - Puppet errors on deployment-docker-mathoid01 is CRITICAL: (Service Check Timed Out) [20:15:50] PROBLEM - Puppet errors on integration-puppetmaster01 is CRITICAL: (Service Check Timed Out) [20:15:50] PROBLEM - Puppet errors on integration-castor03 is CRITICAL: (Service Check Timed Out) [20:15:51] PROBLEM - Puppet staleness on deployment-mcs01 is CRITICAL: (Service Check Timed Out) [20:15:53] PROBLEM - Free space - all mounts on deployment-db06 is CRITICAL: (Service Check Timed Out) [20:15:59] PROBLEM - Puppet errors on integration-slave-docker-1052 is CRITICAL: (Service Check Timed Out) [20:16:02] PROBLEM - Puppet errors on deployment-puppetmaster03 is CRITICAL: (Service Check Timed Out) [20:16:02] PROBLEM - Puppet errors on deployment-prometheus02 is CRITICAL: (Service Check Timed Out) [20:16:02] PROBLEM - Puppet staleness on integration-r-lang-01 is CRITICAL: (Service Check Timed Out) [20:16:04] PROBLEM - Puppet staleness on deployment-aqs03 is CRITICAL: (Service Check Timed Out) [20:16:10] PROBLEM - Puppet staleness on deployment-elastic05 is CRITICAL: (Service Check Timed Out) [20:16:10] PROBLEM - Puppet errors on deployment-webperf11 is CRITICAL: (Service Check Timed Out) [20:16:13] PROBLEM - Free space - all mounts on deployment-puppetmaster03 is CRITICAL: (Service Check Timed Out) [20:16:14] PROBLEM - Puppet errors on integration-slave-docker-1054 is CRITICAL: (Service Check Timed Out) [20:16:17] PROBLEM - Free space - all mounts on deployment-kafka-jumbo-1 is CRITICAL: (Service Check Timed Out) [20:16:17] PROBLEM - Free space - all mounts on integration-slave-jessie-1001 is CRITICAL: (Service Check Timed Out) [20:16:17] PROBLEM - Puppet errors on deployment-sessionstore01 is CRITICAL: (Service Check Timed Out) [20:16:17] PROBLEM - Puppet staleness on integration-puppetmaster01 is CRITICAL: (Service Check Timed Out) [20:16:19] PROBLEM - Free space - all mounts on deployment-alex-test is CRITICAL: (Service Check Timed Out) [20:16:20] PROBLEM - Free space - all mounts on deployment-acme-chief04 is CRITICAL: (Service Check Timed Out) [20:16:23] PROBLEM - Puppet errors on deployment-sessionstore02 is CRITICAL: (Service Check Timed Out) [20:16:50] PROBLEM - Mediawiki Error Rate on graphite-labs is CRITICAL: (Service Check Timed Out) [20:16:50] PROBLEM - Free space - all mounts on integration-slave-docker-1040 is CRITICAL: (Service Check Timed Out) [20:16:50] PROBLEM - Puppet errors on deployment-ms-fe03 is CRITICAL: (Service Check Timed Out) [20:16:50] PROBLEM - Puppet staleness on deployment-pdfrender02 is CRITICAL: (Service Check Timed Out) [20:16:53] PROBLEM - Puppet staleness on deployment-sca04 is CRITICAL: (Service Check Timed Out) [20:16:53] PROBLEM - Puppet staleness on integration-slave-docker-1043 is CRITICAL: (Service Check Timed Out) [20:16:53] PROBLEM - Free space - all mounts on deployment-cumin02 is CRITICAL: (Service Check Timed Out) [20:16:53] PROBLEM - Free space - all mounts on deployment-memc05 is CRITICAL: (Service Check Timed Out) [20:16:54] PROBLEM - Free space - all mounts on deployment-chromium01 is CRITICAL: (Service Check Timed Out) [20:16:56] PROBLEM - Puppet errors on deployment-elastic05 is CRITICAL: (Service Check Timed Out) [20:16:58] PROBLEM - Puppet errors on deployment-sca04 is CRITICAL: (Service Check Timed Out) [20:16:59] PROBLEM - Free space - all mounts on deployment-deploy02 is CRITICAL: (Service Check Timed Out) [20:16:59] PROBLEM - Puppet errors on deployment-cache-upload05 is CRITICAL: (Service Check Timed Out) [20:17:02] PROBLEM - Puppet staleness on deployment-memc06 is CRITICAL: (Service Check Timed Out) [20:17:02] PROBLEM - Puppet errors on deployment-snapshot01 is CRITICAL: (Service Check Timed Out) [20:17:02] PROBLEM - Free space - all mounts on deployment-eventlog05 is CRITICAL: (Service Check Timed Out) [20:17:07] PROBLEM - Puppet errors on deployment-sca01 is CRITICAL: (Service Check Timed Out) [20:17:07] PROBLEM - Puppet staleness on deployment-mediawiki-09 is CRITICAL: (Service Check Timed Out) [20:17:07] PROBLEM - Puppet staleness on deployment-etcd-01 is CRITICAL: (Service Check Timed Out) [20:17:07] PROBLEM - Puppet errors on deployment-mediawiki-07 is CRITICAL: (Service Check Timed Out) [20:17:09] PROBLEM - Free space - all mounts on deployment-chromium02 is CRITICAL: (Service Check Timed Out) [20:17:12] PROBLEM - Free space - all mounts on deployment-elastic07 is CRITICAL: (Service Check Timed Out) [20:17:12] PROBLEM - Puppet errors on deployment-hadoop-test-3 is CRITICAL: (Service Check Timed Out) [20:17:13] PROBLEM - Puppet staleness on deployment-hadoop-test-2 is CRITICAL: (Service Check Timed Out) [20:17:13] PROBLEM - Free space - all mounts on deployment-fluorine02 is CRITICAL: (Service Check Timed Out) [20:17:16] PROBLEM - Puppet staleness on saucelabs-01 is CRITICAL: (Service Check Timed Out) [20:17:17] PROBLEM - Puppet staleness on deployment-kafka-main-1 is CRITICAL: (Service Check Timed Out) [20:17:17] PROBLEM - Puppet staleness on deployment-elastic07 is CRITICAL: (Service Check Timed Out) [20:17:18] PROBLEM - Free space - all mounts on deployment-prometheus02 is CRITICAL: (Service Check Timed Out) [20:17:19] PROBLEM - Free space - all mounts on integration-trigger-01 is CRITICAL: (Service Check Timed Out) [20:17:20] PROBLEM - Puppet errors on deployment-urldownloader02 is CRITICAL: (Service Check Timed Out) [20:17:20] PROBLEM - Puppet staleness on deployment-eventgate-1 is CRITICAL: (Service Check Timed Out) [20:17:22] PROBLEM - Puppet staleness on deployment-cache-text05 is CRITICAL: (Service Check Timed Out) [20:17:24] PROBLEM - Free space - all mounts on deployment-mx02 is CRITICAL: (Service Check Timed Out) [20:17:25] PROBLEM - Free space - all mounts on deployment-acme-chief03 is CRITICAL: (Service Check Timed Out) [20:17:25] PROBLEM - Puppet errors on deployment-changeprop is CRITICAL: (Service Check Timed Out) [20:17:26] PROBLEM - Puppet staleness on deployment-wikifeeds01 is CRITICAL: (Service Check Timed Out) [20:17:27] PROBLEM - Puppet staleness on deployment-chromium02 is CRITICAL: (Service Check Timed Out) [20:17:27] PROBLEM - Puppet staleness on deployment-acme-chief04 is CRITICAL: (Service Check Timed Out) [20:17:29] PROBLEM - Puppet errors on deployment-sca02 is CRITICAL: (Service Check Timed Out) [20:17:29] PROBLEM - Free space - all mounts on deployment-sessionstore01 is CRITICAL: (Service Check Timed Out) [20:17:33] PROBLEM - Puppet errors on integration-slave-docker-1051 is CRITICAL: (Service Check Timed Out) [20:17:34] PROBLEM - Puppet staleness on deployment-acme-chief03 is CRITICAL: (Service Check Timed Out) [20:20:01] ohai shinken spam [20:20:37] RECOVERY - Puppet errors on deployment-docker-mathoid01 is OK: OK: Less than 1.00% above the threshold [2.0] [20:20:41] RECOVERY - Puppet errors on integration-puppetmaster01 is OK: OK: Less than 1.00% above the threshold [2.0] [20:20:42] RECOVERY - Puppet errors on integration-castor03 is OK: OK: Less than 1.00% above the threshold [2.0] [20:20:43] RECOVERY - Free space - all mounts on deployment-db06 is OK: OK: deployment-prep.deployment-db06.diskspace._srv.byte_percentfree (More than half of the datapoints are undefined) deployment-prep.deployment-db06.diskspace.root.byte_percentfree (More than half of the datapoints are undefined) [20:20:49] RECOVERY - Puppet errors on integration-slave-docker-1052 is OK: OK: Less than 1.00% above the threshold [2.0] [20:20:53] RECOVERY - Puppet errors on deployment-puppetmaster03 is OK: OK: Less than 1.00% above the threshold [2.0] [20:20:53] RECOVERY - Puppet errors on deployment-prometheus02 is OK: OK: Less than 1.00% above the threshold [2.0] [20:20:53] RECOVERY - Puppet staleness on deployment-aqs03 is OK: OK: Less than 1.00% above the threshold [3600.0] [20:21:03] RECOVERY - Puppet errors on deployment-webperf11 is OK: OK: Less than 1.00% above the threshold [2.0] [20:21:03] RECOVERY - Free space - all mounts on deployment-puppetmaster03 is OK: OK: deployment-prep.deployment-puppetmaster03.diskspace.root.byte_percentfree (More than half of the datapoints are undefined) [20:21:03] RECOVERY - Puppet errors on integration-slave-docker-1054 is OK: OK: Less than 1.00% above the threshold [2.0] [20:21:07] RECOVERY - Free space - all mounts on integration-slave-jessie-1001 is OK: OK: integration.integration-slave-jessie-1001.diskspace._mnt.byte_percentfree (No valid datapoints found) integration.integration-slave-jessie-1001.diskspace.root.byte_percentfree (More than half of the datapoints are undefined) integration.integration-slave-jessie-1001.diskspace._srv.byte_percentfree (More than half of the datapoints are undefined) [20:21:07] RECOVERY - Puppet errors on deployment-sessionstore01 is OK: OK: Less than 1.00% above the threshold [2.0] [20:21:10] RECOVERY - Free space - all mounts on deployment-kafka-jumbo-1 is OK: OK: deployment-prep.deployment-kafka-jumbo-1.diskspace.root.byte_percentfree (More than half of the datapoints are undefined) deployment-prep.deployment-kafka-jumbo-1.diskspace._mnt_kafka.byte_percentfree (No valid datapoints found) deployment-prep.deployment-kafka-jumbo-1.diskspace._srv_kafka.byte_percentfree (More than half of the datapoints are undefined) [20:21:12] RECOVERY - Puppet errors on deployment-sessionstore02 is OK: OK: Less than 1.00% above the threshold [2.0] [20:21:13] RECOVERY - Free space - all mounts on deployment-alex-test is OK: OK: deployment-prep.deployment-alex-test.diskspace.root.byte_percentfree (More than half of the datapoints are undefined) [20:21:13] RECOVERY - Free space - all mounts on deployment-acme-chief04 is OK: OK: deployment-prep.deployment-acme-chief04.diskspace.root.byte_percentfree (More than half of the datapoints are undefined) [20:21:42] RECOVERY - Mediawiki Error Rate on graphite-labs is OK: OK: Less than 1.00% above the threshold [1.0] [20:21:43] RECOVERY - Free space - all mounts on integration-slave-docker-1040 is OK: OK: All targets OK [20:21:43] RECOVERY - Puppet errors on deployment-ms-fe03 is OK: OK: Less than 1.00% above the threshold [2.0] [20:21:43] RECOVERY - Puppet staleness on deployment-pdfrender02 is OK: OK: Less than 1.00% above the threshold [3600.0] [20:21:44] RECOVERY - Free space - all mounts on deployment-cumin02 is OK: OK: deployment-prep.deployment-cumin02.diskspace.root.byte_percentfree (More than half of the datapoints are undefined) [20:21:44] RECOVERY - Free space - all mounts on deployment-memc05 is OK: OK: All targets OK [20:21:44] RECOVERY - Puppet staleness on integration-slave-docker-1043 is OK: OK: Less than 1.00% above the threshold [3600.0] [20:21:46] RECOVERY - Puppet errors on deployment-sca04 is OK: OK: Less than 1.00% above the threshold [2.0] [20:21:46] RECOVERY - Free space - all mounts on deployment-chromium01 is OK: OK: deployment-prep.deployment-chromium01.diskspace.root.byte_percentfree (More than half of the datapoints are undefined) [20:21:46] RECOVERY - Puppet errors on deployment-elastic05 is OK: OK: Less than 1.00% above the threshold [2.0] [20:21:48] RECOVERY - Puppet errors on deployment-cache-upload05 is OK: OK: Less than 1.00% above the threshold [2.0] [20:21:48] RECOVERY - Free space - all mounts on deployment-deploy02 is OK: OK: deployment-prep.deployment-deploy02.diskspace._srv.byte_percentfree (More than half of the datapoints are undefined) deployment-prep.deployment-deploy02.diskspace.root.byte_percentfree (More than half of the datapoints are undefined) [20:21:52] RECOVERY - Free space - all mounts on deployment-eventlog05 is OK: OK: deployment-prep.deployment-eventlog05.diskspace._var_lib_mysql.byte_percentfree (No valid datapoints found) [20:21:52] RECOVERY - Puppet staleness on deployment-memc06 is OK: OK: Less than 1.00% above the threshold [3600.0] [20:21:55] RECOVERY - Puppet errors on deployment-snapshot01 is OK: OK: Less than 1.00% above the threshold [2.0] [20:21:58] RECOVERY - Puppet errors on deployment-sca01 is OK: OK: Less than 1.00% above the threshold [2.0] [20:21:58] RECOVERY - Puppet errors on deployment-mediawiki-07 is OK: OK: Less than 1.00% above the threshold [2.0] [20:21:58] RECOVERY - Free space - all mounts on deployment-chromium02 is OK: OK: All targets OK [20:22:01] RECOVERY - Free space - all mounts on deployment-elastic07 is OK: OK: deployment-prep.deployment-elastic07.diskspace._var_lib_elasticsearch.byte_percentfree (No valid datapoints found) deployment-prep.deployment-elastic07.diskspace.root.byte_percentfree (More than half of the datapoints are undefined) deployment-prep.deployment-elastic07.diskspace._srv.byte_percentfree (More than half of the datapoints are undefined) deployment-pre [20:22:01] astic07.diskspace._var_log.byte_percentfree (No valid datapoints found) [20:22:02] RECOVERY - Puppet errors on deployment-hadoop-test-3 is OK: OK: Less than 1.00% above the threshold [2.0] [20:22:02] RECOVERY - Puppet staleness on deployment-hadoop-test-2 is OK: OK: Less than 1.00% above the threshold [3600.0] [20:22:04] RECOVERY - Free space - all mounts on deployment-fluorine02 is OK: OK: deployment-prep.deployment-fluorine02.diskspace.root.byte_percentfree (More than half of the datapoints are undefined) deployment-prep.deployment-fluorine02.diskspace._srv.byte_percentfree (More than half of the datapoints are undefined) [20:22:07] RECOVERY - Free space - all mounts on deployment-prometheus02 is OK: OK: deployment-prep.deployment-prometheus02.diskspace.root.byte_percentfree (More than half of the datapoints are undefined) deployment-prep.deployment-prometheus02.diskspace._srv.byte_percentfree (More than half of the datapoints are undefined) [20:22:07] RECOVERY - Free space - all mounts on integration-trigger-01 is OK: OK: integration.integration-trigger-01.diskspace._srv.byte_percentfree (No valid datapoints found) integration.integration-trigger-01.diskspace.root.byte_percentfree (More than half of the datapoints are undefined) [20:22:09] RECOVERY - Puppet errors on deployment-urldownloader02 is OK: OK: Less than 1.00% above the threshold [2.0] [20:22:11] RECOVERY - Free space - all mounts on deployment-mx02 is OK: OK: deployment-prep.deployment-mx02.diskspace.root.byte_percentfree (More than half of the datapoints are undefined) [20:22:11] RECOVERY - Puppet errors on deployment-changeprop is OK: OK: Less than 1.00% above the threshold [2.0] [20:22:14] RECOVERY - Free space - all mounts on deployment-acme-chief03 is OK: OK: deployment-prep.deployment-acme-chief03.diskspace.root.byte_percentfree (More than half of the datapoints are undefined) [20:22:21] RECOVERY - Puppet errors on deployment-sca02 is OK: OK: Less than 1.00% above the threshold [2.0] [20:22:21] RECOVERY - Free space - all mounts on deployment-sessionstore01 is OK: OK: deployment-prep.deployment-sessionstore01.diskspace.root.byte_percentfree (More than half of the datapoints are undefined) [20:22:23] RECOVERY - Puppet errors on integration-slave-docker-1051 is OK: OK: Less than 1.00% above the threshold [2.0] [20:22:28] RECOVERY - Free space - all mounts on integration-slave-docker-1041 is OK: OK: integration.integration-slave-docker-1041.diskspace.root.byte_percentfree (More than half of the datapoints are undefined) integration.integration-slave-docker-1041.diskspace._srv.byte_percentfree (More than half of the datapoints are undefined) integration.integration-slave-docker-1041.diskspace._var_lib_docker.byte_percentfree (More than half of the da [20:22:29] defined) [20:22:30] RECOVERY - Puppet errors on deployment-docker-cxserver01 is OK: OK: Less than 1.00% above the threshold [2.0] [20:22:32] RECOVERY - Puppet errors on deployment-memc05 is OK: OK: Less than 1.00% above the threshold [2.0] [20:22:34] RECOVERY - Puppet errors on deployment-restbase02 is OK: OK: Less than 1.00% above the threshold [2.0] [20:22:35] RECOVERY - Puppet errors on deployment-mx02 is OK: OK: Less than 1.00% above the threshold [2.0] [20:22:35] RECOVERY - Puppet errors on deployment-eventlog05 is OK: OK: Less than 1.00% above the threshold [2.0] [20:22:38] RECOVERY - Puppet errors on deployment-acme-chief04 is OK: OK: Less than 1.00% above the threshold [2.0] [20:22:39] 10Continuous-Integration-Config, 10phan-taint-check-plugin, 10Patch-For-Review: Upgrade php-ast to 1.0.1 in CI containers - https://phabricator.wikimedia.org/T218719 (10Daimona) >>! In T218719#5304507, @Jdforrester-WMF wrote: >>>! In T218719#5300491, @Daimona wrote: >>>>! In T218719#5300479, @Jdforrester-WMF... [20:22:39] RECOVERY - Puppet errors on deployment-deploy01 is OK: OK: Less than 1.00% above the threshold [2.0] [20:22:42] RECOVERY - Free space - all mounts on integration-slave-docker-1052 is OK: OK: integration.integration-slave-docker-1052.diskspace._srv.byte_percentfree (More than half of the datapoints are undefined) integration.integration-slave-docker-1052.diskspace._var_lib_docker.byte_percentfree (More than half of the datapoints are undefined) integration.integration-slave-docker-1052.diskspace.root.byte_percentfree (More than half of the da [20:22:43] defined) [20:22:46] RECOVERY - Puppet errors on deployment-chromium01 is OK: OK: Less than 1.00% above the threshold [2.0] [20:22:46] RECOVERY - Puppet errors on integration-cumin is OK: OK: Less than 1.00% above the threshold [2.0] [20:22:47] RECOVERY - Free space - all mounts on deployment-ores01 is OK: OK: deployment-prep.deployment-ores01.diskspace._srv.byte_percentfree (More than half of the datapoints are undefined) deployment-prep.deployment-ores01.diskspace.root.byte_percentfree (More than half of the datapoints are undefined) [20:22:55] RECOVERY - Free space - all mounts on integration-slave-jessie-1002 is OK: OK: integration.integration-slave-jessie-1002.diskspace.root.byte_percentfree (More than half of the datapoints are undefined) integration.integration-slave-jessie-1002.diskspace._srv.byte_percentfree (More than half of the datapoints are undefined) integration.integration-slave-jessie-1002.diskspace._mnt.byte_percentfree (No valid datapoints found) [20:22:58] RECOVERY - Puppet errors on deployment-kafka-main-2 is OK: OK: Less than 1.00% above the threshold [2.0] [20:23:01] RECOVERY - Free space - all mounts on integration-slave-docker-1050 is OK: OK: integration.integration-slave-docker-1050.diskspace._var_lib_docker.byte_percentfree (More than half of the datapoints are undefined) integration.integration-slave-docker-1050.diskspace._srv.byte_percentfree (More than half of the datapoints are undefined) integration.integration-slave-docker-1050.diskspace.root.byte_percentfree (More than half of the da [20:23:01] defined) [20:23:01] RECOVERY - Free space - all mounts on integration-slave-docker-1054 is OK: OK: integration.integration-slave-docker-1054.diskspace.root.byte_percentfree (More than half of the datapoints are undefined) integration.integration-slave-docker-1054.diskspace._var_lib_docker.byte_percentfree (More than half of the datapoints are undefined) integration.integration-slave-docker-1054.diskspace._srv.byte_percentfree (More than half of the da [20:23:01] defined) [20:23:02] sorry for the shinken spam folks. We rebooted a host for kernel updates and did not expect ti to make all the alarms go off. [20:23:03] RECOVERY - Puppet errors on deployment-chromium02 is OK: OK: Less than 1.00% above the threshold [2.0] [20:23:06] RECOVERY - Free space - all mounts on deployment-ms-be05 is OK: OK: All targets OK [20:23:06] RECOVERY - Free space - all mounts on deployment-imagescaler02 is OK: OK: deployment-prep.deployment-imagescaler02.diskspace.root.byte_percentfree (More than half of the datapoints are undefined) [20:23:14] RECOVERY - Free space - all mounts on deployment-zookeeper02 is OK: OK: deployment-prep.deployment-zookeeper02.diskspace.root.byte_percentfree (More than half of the datapoints are undefined) [20:23:14] RECOVERY - Free space - all mounts on deployment-webperf11 is OK: OK: deployment-prep.deployment-webperf11.diskspace.root.byte_percentfree (More than half of the datapoints are undefined) [20:23:14] RECOVERY - Free space - all mounts on deployment-mwmaint01 is OK: OK: deployment-prep.deployment-mwmaint01.diskspace.root.byte_percentfree (More than half of the datapoints are undefined) [20:23:24] RECOVERY - Free space - all mounts on integration-slave-docker-1059 is OK: OK: All targets OK [20:23:27] RECOVERY - Free space - all mounts on integration-slave-docker-1051 is OK: OK: All targets OK [20:23:35] RECOVERY - Free space - all mounts on deployment-elastic05 is OK: OK: deployment-prep.deployment-elastic05.diskspace._var_log.byte_percentfree (No valid datapoints found) deployment-prep.deployment-elastic05.diskspace._var_lib_elasticsearch.byte_percentfree (No valid datapoints found) [20:23:38] RECOVERY - Puppet errors on deployment-logstash03 is OK: OK: Less than 1.00% above the threshold [2.0] [20:23:38] RECOVERY - Puppet staleness on integration-agent-docker-1001 is OK: OK: Less than 1.00% above the threshold [3600.0] [20:24:01] RECOVERY - Puppet errors on deployment-mediawiki-09 is OK: OK: Less than 1.00% above the threshold [2.0] [20:24:04] wb [20:24:14] bd808: s'ok [20:25:36] RECOVERY - Puppet staleness on deployment-fluorine02 is OK: OK: Less than 1.00% above the threshold [3600.0] [20:25:42] RECOVERY - Puppet staleness on deployment-mcs01 is OK: OK: Less than 1.00% above the threshold [3600.0] [20:25:54] RECOVERY - Puppet staleness on integration-r-lang-01 is OK: OK: Less than 1.00% above the threshold [3600.0] [20:26:00] RECOVERY - Puppet staleness on deployment-elastic05 is OK: OK: Less than 1.00% above the threshold [3600.0] [20:26:08] RECOVERY - Puppet staleness on integration-puppetmaster01 is OK: OK: Less than 1.00% above the threshold [3600.0] [20:26:43] RECOVERY - Puppet staleness on deployment-sca04 is OK: OK: Less than 1.00% above the threshold [3600.0] [20:27:01] RECOVERY - Puppet staleness on deployment-mediawiki-09 is OK: OK: Less than 1.00% above the threshold [3600.0] [20:27:01] RECOVERY - Puppet staleness on deployment-etcd-01 is OK: OK: Less than 1.00% above the threshold [3600.0] [20:27:04] RECOVERY - Puppet staleness on deployment-kafka-main-1 is OK: OK: Less than 1.00% above the threshold [3600.0] [20:27:04] RECOVERY - Puppet staleness on saucelabs-01 is OK: OK: Less than 1.00% above the threshold [3600.0] [20:27:04] RECOVERY - Puppet staleness on deployment-elastic07 is OK: OK: Less than 1.00% above the threshold [3600.0] [20:27:08] RECOVERY - Puppet staleness on deployment-cache-text05 is OK: OK: Less than 1.00% above the threshold [3600.0] [20:27:09] RECOVERY - Puppet staleness on deployment-eventgate-1 is OK: OK: Less than 1.00% above the threshold [3600.0] [20:27:14] RECOVERY - Puppet staleness on deployment-wikifeeds01 is OK: OK: Less than 1.00% above the threshold [3600.0] [20:27:14] RECOVERY - Puppet staleness on deployment-chromium02 is OK: OK: Less than 1.00% above the threshold [3600.0] [20:27:17] RECOVERY - Puppet staleness on deployment-acme-chief04 is OK: OK: Less than 1.00% above the threshold [3600.0] [20:27:22] RECOVERY - Puppet staleness on deployment-acme-chief03 is OK: OK: Less than 1.00% above the threshold [3600.0] [20:27:29] RECOVERY - Puppet staleness on deployment-db06 is OK: OK: Less than 1.00% above the threshold [3600.0] [20:27:32] RECOVERY - Puppet staleness on deployment-puppetdb02 is OK: OK: Less than 1.00% above the threshold [3600.0] [20:27:34] RECOVERY - Puppet staleness on integration-slave-docker-1050 is OK: OK: Less than 1.00% above the threshold [3600.0] [20:27:36] RECOVERY - Puppet staleness on deployment-ms-fe03 is OK: OK: Less than 1.00% above the threshold [3600.0] [20:27:49] RECOVERY - Puppet staleness on deployment-cumin02 is OK: OK: Less than 1.00% above the threshold [3600.0] [20:27:50] RECOVERY - Puppet staleness on deployment-logstash2 is OK: OK: Less than 1.00% above the threshold [3600.0] [20:27:51] RECOVERY - Puppet staleness on deployment-sentry01 is OK: OK: Less than 1.00% above the threshold [3600.0] [20:27:56] RECOVERY - Puppet staleness on deployment-sessionstore01 is OK: OK: Less than 1.00% above the threshold [3600.0] [20:27:56] RECOVERY - Puppet staleness on deployment-mwmaint01 is OK: OK: Less than 1.00% above the threshold [3600.0] [20:28:05] RECOVERY - Puppet staleness on deployment-maps04 is OK: OK: Less than 1.00% above the threshold [3600.0] [20:28:10] RECOVERY - Puppet staleness on deployment-puppetmaster03 is OK: OK: Less than 1.00% above the threshold [3600.0] [20:28:10] RECOVERY - Puppet staleness on webperformance is OK: OK: Less than 1.00% above the threshold [3600.0] [20:28:11] RECOVERY - Puppet staleness on deployment-sessionstore02 is OK: OK: Less than 1.00% above the threshold [3600.0] [20:28:27] RECOVERY - Puppet staleness on deployment-poolcounter05 is OK: OK: Less than 1.00% above the threshold [3600.0] [20:29:10] 10Scap: release a scap that contains I85a2161 (Remove functionality to talk to conftool) - https://phabricator.wikimedia.org/T227225 (10CDanis) [20:30:19] paladox: looking at https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/509172/5/modules/profile/manifests/gerrit/server.pp after merging https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/510625/ [20:30:57] yup [20:30:59] paladox: wondering if we want to send gerrit.log and gerrit.json or really just gerrit.json.. yes, we don't want to send files called _log anymore [20:31:07] but what about gerrit.log [20:31:18] that used to be error_log :) [20:31:24] We want to send gerrit.json. [20:31:26] but was more than errors [20:31:46] since sending both gerrit.log and gerritjson would result in duplicates in logstash [20:32:03] do we get all the warnings and stuff i see now in gerrit.log also from the .json ? [20:32:12] yup [20:32:15] 10Scap: release a scap that contains I85a2161 (Remove functionality to talk to conftool) - https://phabricator.wikimedia.org/T227225 (10thcipriani) p:05Triage→03Normal [20:32:16] ok, great! [20:34:07] thcipriani: so you are not surprised. on next restart of gerrit there should be a gerrit.log now . that used to be what was error_log but the name was misleading [20:34:19] it doesnt have just errors [20:34:49] mutante: sounds good and makes sense [20:34:51] and the second thing is where you gave me a +1 (thanks) .. send only the json file to logstash [20:35:09] and confirmed my paladox that should be enough.. doing that [20:35:37] yup. [20:41:53] we are not sure if sending the sshd logs to logstash is desired [20:43:41] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team, 10Release-Engineering-Team-TODO, 10Epic: Meeting: Automatic deployment of backend services on beta cluster - https://phabricator.wikimedia.org/T100099 (10kchapman) [20:46:34] thcipriani ^^ [20:48:01] seems like it would be desired; although uses a completely different log class, so unclear how to stuff them into gerrit.json is where we left off IIRC [20:54:02] thcipriani i think we will have to export both log files (no easy way of putting the ssh log in gerrit.json) [20:54:21] both as in ssh_log and gerrit.json [20:56:12] yeah, that seems like the only way of getting them both in logstash, although the sshd logs would be mostly unstructured in that case was my understanding. [21:01:28] i wonder if sshd logs need a 'startmsg_regex' parameter in rsyslog::input::file or not [21:01:58] like gerrit-multine had [21:03:06] 10Phabricator, 10Project-Admins: Create User-RhinosF1 Project on Phab - https://phabricator.wikimedia.org/T227227 (10RhinosF1) [21:03:42] 10Phabricator, 10Project-Admins: Create User-RhinosF1 Project on Phab - https://phabricator.wikimedia.org/T227227 (10RhinosF1) p:05Triage→03Normal [21:13:40] mutante i *think* so [21:15:19] that would be my assumption as well [21:15:25] paladox: if so.. it could be tricky to write one [21:15:34] but should be out there somewhere already [21:17:13] mutante it should the same as what we used for error_log [21:17:53] oh really, well that is of course easier [21:24:14] 10Project-Admins: Create User-RhinosF1 Project on Phab - https://phabricator.wikimedia.org/T227227 (10Aklapper) [21:29:20] 10Continuous-Integration-Config: Move all CI generic tasks from PHP70 to PHP72 - https://phabricator.wikimedia.org/T225457 (10Jdforrester-WMF) [21:29:22] 10Continuous-Integration-Config, 10Patch-For-Review: Run phan secheck on PHP 7.2, not PHP 7.0 - https://phabricator.wikimedia.org/T226420 (10Jdforrester-WMF) 05Open→03Stalled Yeah, I thought it'd be easy too. :-( [22:09:49] thcipriani mutante https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/520644/ [22:10:20] paladox: https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/508391/3/modules/gerrit/templates/log4j.xml.erb@14 :) [22:10:29] amend before we forgot it :) [22:10:52] ah, i see paladox.. yea. per IRC :) [22:10:56] yup [22:11:24] so, now we can send gc/delete/replication to the gerrit.log/gerrit.json :) (i'll do a follow up commit) [22:11:42] cool [22:16:16] confirmed /var/log/gerrit/httpd_log is empty [22:16:40] i knew we had it disabled, just double checking because of the patch [22:22:48] paladox: merged [22:23:09] mutante thanks! [22:23:14] i've updated the other change too [22:29:11] 10Release-Engineering-Team-TODO (201907), 10Scap: release a scap that contains I85a2161 (Remove functionality to talk to conftool) - https://phabricator.wikimedia.org/T227225 (10greg) [22:29:29] 10Release-Engineering-Team-TODO (201907), 10Wikimedia-Site-requests, 10WikimediaMessages: Put "shim" code for namespaces, logs, and log i18n into WikimediaMessages so we can undeploy extensions - https://phabricator.wikimedia.org/T222918 (10greg) p:05Triage→03Normal [22:29:31] 10Release-Engineering-Team-TODO (201907), 10Code-Stewardship-Reviews: Code Stewardship Review: OAuth extension - https://phabricator.wikimedia.org/T224919 (10greg) p:05Triage→03Normal [22:29:33] 10Release-Engineering-Team-TODO (201907), 10Code-Stewardship-Reviews: Code Stewardship Review: SpamBlacklist - https://phabricator.wikimedia.org/T224921 (10greg) p:05Triage→03Normal [22:29:35] 10Release-Engineering-Team-TODO (201907), 10Code-Stewardship-Reviews: Code Stewardship Review: Collection Extension - https://phabricator.wikimedia.org/T224922 (10greg) p:05Triage→03Normal [22:29:51] 10Release-Engineering-Team-TODO (201907), 10Developer Productivity, 10local-charts, 10Patch-For-Review: Move local-charts helm charts to a chart repository - https://phabricator.wikimedia.org/T224935 (10greg) p:05Triage→03Normal [22:29:53] 10Continuous-Integration-Config, 10Release-Engineering-Team-TODO (201907), 10AbuseFilter, 10CX-deployments, and 2 others: mediawiki/vendor REL1_* no longer ship dependencies for wmf extensions that are not in the mediawiki tarball - https://phabricator.wikimedia.org/T189560 (10greg) p:05Triage→03Normal [22:32:25] 10Project-Admins: Create User-RhinosF1 Project on Phab - https://phabricator.wikimedia.org/T227227 (10MarcoAurelio) I can create the project, but I am not able to modify the edit policies. It looks only Administrators and perhaps #acl*phabricator people would be able to do it. [22:32:39] 10Gerrit, 10Patch-For-Review: Unable to login to gerrit - https://phabricator.wikimedia.org/T223266 (10Shirayuki) Hello? [22:44:04] 10Gerrit, 10Patch-For-Review: Unable to login to gerrit - https://phabricator.wikimedia.org/T223266 (10Paladox) a:03thcipriani [22:45:05] 10Gerrit, 10Patch-For-Review: Unable to login to gerrit - https://phabricator.wikimedia.org/T223266 (10Dzahn) Hi @Shirayuki, i think @thcipriani can help you but please keep in mind there is a big public holiday in the US tomorrow. [23:13:15] 10Release-Engineering-Team-TODO (201907), 10Scap: release a scap that contains I85a2161 (Remove functionality to talk to conftool) - https://phabricator.wikimedia.org/T227225 (10thcipriani) a:05thcipriani→03CDanis @CDanis I merged in the changes to the `release` branch and pushed up the `debian/3.11.0-1` t... [23:14:15] 10Release-Engineering-Team-TODO (201907), 10Scap: release a scap that contains I85a2161 (Remove functionality to talk to conftool) - https://phabricator.wikimedia.org/T227225 (10CDanis) a:05CDanis→03Volans Wow, that was quick, thanks! Riccardo should have time to do the deploy while I'm on vacation 🙃