[00:43:57] Reedy: blame ori. He ripped the copy-pasta json formatting code out of multiversion. php 5.3 doens't pretty print :( [01:01:51] bd808: s/pretty print// [01:01:57] It doesn't do anything right... [01:40:51] lol [01:40:59] bd808: fwiw, there's a warning I'm seeing a ton of in hhvm.log on fluorine that's not showing up logstash's fatalmonitor dashboard. [01:41:00] grep testwiki wikiversions.json [01:41:03] 1 line of text [01:41:05] FUCK YOU [01:41:16] https://phabricator.wikimedia.org/T105810 [01:52:13] ostriches: we filter out a lot of stuff from logstash fatalmonitor. Check the hhvm dashboard for the unfiltered view [01:53:05] I noticed that earlier today. We may be throwing out more than we should at this point [02:04:39] bd808: I think so [02:04:43] There's a lot that's missing [02:06:04] hhvm dash has no events ... [02:06:13] oh, still spinning lol [02:07:29] bd808: hhvm dash is exactly what I want, if it just had the grouping like fatalmonitor [02:35:38] Project browsertests-WikiLove-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #633: FAILURE in 2 min 37 sec: https://integration.wikimedia.org/ci/job/browsertests-WikiLove-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/633/ [03:15:06] mw2210.codfw.wmnet: l10n merge: 100% (ok: 385; fail: 0; left: 0) [03:15:06] mw2210.codfw.wmnet: l10n merge: 100% (ok: 385; fail: 0; left: 0) [03:15:10] dupe messages [04:16:50] ostriches: https://logstash.wikimedia.org/#/dashboard/elasticsearch/hhvm now has table of top 15 messages [04:18:30] wheee exactly what I needed :D [04:18:32] ty sir [05:00:17] !log created mira.deployment-prep.eqiad.wmflabs to begin testing multi-master scap [05:00:20] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [05:04:49] PROBLEM - Puppet failure on mira is CRITICAL 100.00% of data above the critical threshold [0.0] [06:45:34] RECOVERY - Free space - all mounts on deployment-videoscaler01 is OK All targets OK [08:24:26] (03CR) 10Gilles: "Hashar: ping" [integration/config] - 10https://gerrit.wikimedia.org/r/195021 (owner: 10Krinkle) [08:47:55] 10Continuous-Integration-Infrastructure, 6operations: Phase out lanthanum.eqiad.wmnet - https://phabricator.wikimedia.org/T86658#1453240 (10hashar) [08:50:24] 10Beta-Cluster, 6Release-Engineering, 5Patch-For-Review, 7Regression: Beta cluster logo broken (/static/images/project-logos 404 Not Found) - https://phabricator.wikimedia.org/T105541#1453245 (10hashar) 5Open>3Resolved a:3thcipriani [09:14:54] (03PS2) 10Hashar: tests: consistency of Zuul pipelines declarations [integration/config] - 10https://gerrit.wikimedia.org/r/224517 [09:16:59] (03CR) 10jenkins-bot: [V: 04-1] tests: consistency of Zuul pipelines declarations [integration/config] - 10https://gerrit.wikimedia.org/r/224517 (owner: 10Hashar) [09:17:59] (03PS1) 10Hashar: Trigger tests for ArticleFeedbackv5 [integration/config] - 10https://gerrit.wikimedia.org/r/224759 (https://phabricator.wikimedia.org/T105866) [09:22:25] PROBLEM - Puppet failure on deployment-mx is CRITICAL 100.00% of data above the critical threshold [0.0] [09:22:31] (03CR) 10Hashar: [C: 032] Trigger tests for ArticleFeedbackv5 [integration/config] - 10https://gerrit.wikimedia.org/r/224759 (https://phabricator.wikimedia.org/T105866) (owner: 10Hashar) [09:24:29] (03Merged) 10jenkins-bot: Trigger tests for ArticleFeedbackv5 [integration/config] - 10https://gerrit.wikimedia.org/r/224759 (https://phabricator.wikimedia.org/T105866) (owner: 10Hashar) [09:43:58] Project browsertests-Echo-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce build #547: FAILURE in 6 min 57 sec: https://integration.wikimedia.org/ci/job/browsertests-Echo-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce/547/ [10:01:24] (03PS2) 10Hashar: tests: save a level of indent in test_repos_have_required_jobs [integration/config] - 10https://gerrit.wikimedia.org/r/224418 [10:01:26] (03PS2) 10Hashar: tests: precompile some regex [integration/config] - 10https://gerrit.wikimedia.org/r/224417 [10:01:28] (03PS3) 10Hashar: tests: consistency of Zuul pipelines declarations [integration/config] - 10https://gerrit.wikimedia.org/r/224517 [10:03:36] (03CR) 10jenkins-bot: [V: 04-1] tests: consistency of Zuul pipelines declarations [integration/config] - 10https://gerrit.wikimedia.org/r/224517 (owner: 10Hashar) [10:11:07] (03PS1) 10Hashar: Move some extensions jobs from check-only to check [integration/config] - 10https://gerrit.wikimedia.org/r/224766 [10:13:18] (03CR) 10jenkins-bot: [V: 04-1] Move some extensions jobs from check-only to check [integration/config] - 10https://gerrit.wikimedia.org/r/224766 (owner: 10Hashar) [10:16:05] (03PS2) 10Hashar: Move some extensions jobs from check-only to check [integration/config] - 10https://gerrit.wikimedia.org/r/224766 [10:29:44] (03CR) 10Hashar: "Caught them using the tests in parent change https://gerrit.wikimedia.org/r/224517" [integration/config] - 10https://gerrit.wikimedia.org/r/224766 (owner: 10Hashar) [10:30:03] (03PS3) 10Hashar: Move some extensions jobs from check-only to check [integration/config] - 10https://gerrit.wikimedia.org/r/224766 [10:30:27] (03CR) 10Hashar: [C: 032] Move some extensions jobs from check-only to check [integration/config] - 10https://gerrit.wikimedia.org/r/224766 (owner: 10Hashar) [10:32:29] (03Merged) 10jenkins-bot: Move some extensions jobs from check-only to check [integration/config] - 10https://gerrit.wikimedia.org/r/224766 (owner: 10Hashar) [10:33:43] (03CR) 10Hashar: "Verified with XAnalytics on https://gerrit.wikimedia.org/r/#/c/224761/" [integration/config] - 10https://gerrit.wikimedia.org/r/224766 (owner: 10Hashar) [10:34:17] (03CR) 10Hashar: "recheck" [integration/config] - 10https://gerrit.wikimedia.org/r/224517 (owner: 10Hashar) [11:51:41] (03PS3) 10Hashar: tests: precompile some regex [integration/config] - 10https://gerrit.wikimedia.org/r/224417 [11:56:30] PROBLEM - Free space - all mounts on deployment-videoscaler01 is CRITICAL deployment-prep.deployment-videoscaler01.diskspace._var.byte_percentfree (<20.00%) [12:36:20] (03PS3) 10Hashar: tests: save a level of indent in test_repos_have_required_jobs [integration/config] - 10https://gerrit.wikimedia.org/r/224418 [12:36:29] (03PS4) 10Hashar: tests: consistency of Zuul pipelines declarations [integration/config] - 10https://gerrit.wikimedia.org/r/224517 [12:54:52] Yippee, build fixed! [12:54:53] Project browsertests-GettingStarted-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #534: FIXED in 51 sec: https://integration.wikimedia.org/ci/job/browsertests-GettingStarted-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/534/ [12:56:56] 10Beta-Cluster, 6Release-Engineering: Enable image rotation on beta for testing purposes - https://phabricator.wikimedia.org/T105877#1453679 (10Reedy) 3NEW [13:06:00] Project browsertests-MobileFrontend-en.m.wikipedia.beta.wmflabs.org-linux-chrome-sauce build #717: FAILURE in 33 min: https://integration.wikimedia.org/ci/job/browsertests-MobileFrontend-en.m.wikipedia.beta.wmflabs.org-linux-chrome-sauce/717/ [13:10:58] Project beta-mediawiki-config-update-eqiad build #2902: FAILURE in 0.81 sec: https://integration.wikimedia.org/ci/job/beta-mediawiki-config-update-eqiad/2902/ [14:05:50] (03PS1) 10Hashar: tests: refactor test_repos_have_required_jobs [integration/config] - 10https://gerrit.wikimedia.org/r/224790 [14:05:58] (03Abandoned) 10Hashar: tests: save a level of indent in test_repos_have_required_jobs [integration/config] - 10https://gerrit.wikimedia.org/r/224418 (owner: 10Hashar) [14:07:28] (03CR) 10jenkins-bot: [V: 04-1] tests: refactor test_repos_have_required_jobs [integration/config] - 10https://gerrit.wikimedia.org/r/224790 (owner: 10Hashar) [14:14:20] (03PS2) 10Hashar: tests: refactor test_repos_have_required_jobs [integration/config] - 10https://gerrit.wikimedia.org/r/224790 [14:15:29] (03CR) 10Hashar: "That is an obvious/low hanging fruit. I eventually refactored the whole test with child change https://gerrit.wikimedia.org/r/224790" [integration/config] - 10https://gerrit.wikimedia.org/r/224417 (owner: 10Hashar) [14:15:39] (03CR) 10jenkins-bot: [V: 04-1] tests: refactor test_repos_have_required_jobs [integration/config] - 10https://gerrit.wikimedia.org/r/224790 (owner: 10Hashar) [14:16:45] (03PS5) 10Hashar: tests: consistency of Zuul pipelines declarations [integration/config] - 10https://gerrit.wikimedia.org/r/224517 [14:16:58] (03CR) 10Hashar: "Rebased on top of https://gerrit.wikimedia.org/r/#/c/224790/2" [integration/config] - 10https://gerrit.wikimedia.org/r/224517 (owner: 10Hashar) [14:18:18] (03CR) 10jenkins-bot: [V: 04-1] tests: consistency of Zuul pipelines declarations [integration/config] - 10https://gerrit.wikimedia.org/r/224517 (owner: 10Hashar) [14:23:36] (03PS3) 10Hashar: tests: refactor test_repos_have_required_jobs [integration/config] - 10https://gerrit.wikimedia.org/r/224790 [14:25:24] (03CR) 10Hashar: "PS3 removes some leftover profiling calls :D The specific test runs way faster now." [integration/config] - 10https://gerrit.wikimedia.org/r/224790 (owner: 10Hashar) [14:25:38] (03PS6) 10Hashar: tests: consistency of Zuul pipelines declarations [integration/config] - 10https://gerrit.wikimedia.org/r/224517 [14:31:38] (03PS1) 10Hashar: tests: increase nosetests verbosity [integration/config] - 10https://gerrit.wikimedia.org/r/224797 [14:32:44] (03CR) 10Hashar: "See the console output https://integration.wikimedia.org/ci/job/tox-py27/2083/console :)" [integration/config] - 10https://gerrit.wikimedia.org/r/224797 (owner: 10Hashar) [14:34:22] Yippee, build fixed! [14:34:22] Project browsertests-Echo-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #565: FIXED in 8 min 20 sec: https://integration.wikimedia.org/ci/job/browsertests-Echo-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/565/ [14:36:12] marxarelli: I have some usb sticks that I found at home. I should get the new image from you to put on them. [14:39:15] !log mkdir mira.deployment-prep:/home/l10nupdate because puppet's managehome flag doesn't seem to be doing that :( [14:39:18] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [14:41:29] bd808: yup yup [14:43:45] Yippee, build fixed! [14:43:46] Project browsertests-Wikidata-SmokeTests-linux-firefox-sauce build #314: FIXED in 26 min: https://integration.wikimedia.org/ci/job/browsertests-Wikidata-SmokeTests-linux-firefox-sauce/314/ [15:19:05] (03PS1) 10Hashar: Phase out mediawiki-core-whitespaces [integration/config] - 10https://gerrit.wikimedia.org/r/224804 [15:23:04] (03CR) 10Hashar: [C: 032] Phase out mediawiki-core-whitespaces [integration/config] - 10https://gerrit.wikimedia.org/r/224804 (owner: 10Hashar) [15:23:28] PROBLEM - Puppet failure on deployment-cache-bits01 is CRITICAL 100.00% of data above the critical threshold [0.0] [15:25:19] (03Merged) 10jenkins-bot: Phase out mediawiki-core-whitespaces [integration/config] - 10https://gerrit.wikimedia.org/r/224804 (owner: 10Hashar) [15:33:42] !log root (/) is full on deployment-bastion, trying to figure out why [15:33:46] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [15:35:50] (03PS1) 10Sbisson: [WIP] Support pages with underscore like User_talk [selenium] - 10https://gerrit.wikimedia.org/r/224809 [15:36:54] (03CR) 10jenkins-bot: [V: 04-1] [WIP] Support pages with underscore like User_talk [selenium] - 10https://gerrit.wikimedia.org/r/224809 (owner: 10Sbisson) [15:39:21] hi there! [15:40:04] I've changed a test in mediawiki-selenium and I need help understanding why it fails and how to make it pass. See: https://gerrit.wikimedia.org/r/224809 [15:41:48] bd808: /tmp is 4.7G, for one [15:43:23] demon@deployment-bastion:/tmp$ sudo du -a /tmp | sort -n -r | head -n 10 [15:43:23] 4871364 /tmp [15:43:23] 2592332 /tmp/scap_l10n_3192945711 [15:43:25] 2276260 /tmp/scap_l10n_3917376305 [15:43:27] 5924 /tmp/scap_l10n_3917376305/ml.l10n.php [15:43:29] bd808: lol ^ [15:43:31] 2 files [15:51:19] (03PS1) 10Hashar: Remove some uses of Gerrit replications [integration/config] - 10https://gerrit.wikimedia.org/r/224814 [16:09:34] (03PS12) 10Paladox: Add jsonlint check for MaintenanceShell [integration/config] - 10https://gerrit.wikimedia.org/r/222592 [16:09:44] (03PS13) 10Paladox: Add jsonlint check for MaintenanceShell [integration/config] - 10https://gerrit.wikimedia.org/r/222592 [16:10:41] !log sudo rm -rf /tmp/scap_l10n_* on deployment-bastion [16:10:45] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [16:12:22] (03PS1) 10Hashar: Remove uses of Gerrit replicas in JJB defaults [integration/config] - 10https://gerrit.wikimedia.org/r/224818 [16:14:14] (03PS1) 10Paladox: Use npm for CollapsibleVector [integration/config] - 10https://gerrit.wikimedia.org/r/224819 [16:14:32] Yippee, build fixed! [16:14:32] Project beta-code-update-eqiad build #64085: FIXED in 1 min 31 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/64085/ [16:16:50] (03PS2) 10Paladox: Use npm for CollapsibleVector [integration/config] - 10https://gerrit.wikimedia.org/r/224819 [16:17:44] (03PS2) 10Hashar: Remove some uses of Gerrit replications [integration/config] - 10https://gerrit.wikimedia.org/r/224814 [16:17:46] (03PS2) 10Hashar: Remove uses of Gerrit replicas in JJB defaults [integration/config] - 10https://gerrit.wikimedia.org/r/224818 [16:17:48] (03PS1) 10Hashar: Drop unused use-zuul and git-localonly-zuul [integration/config] - 10https://gerrit.wikimedia.org/r/224821 [16:18:01] 10Continuous-Integration-Infrastructure, 6operations: Phase out lanthanum.eqiad.wmnet - https://phabricator.wikimedia.org/T86658#1454281 (10RobH) a:5RobH>3Cmjohnson The following have been completed: * merge @dzahn's dns change and push live * decom from palladium: puppet keys, salt keys, puppetstoreddb *... [16:18:05] (03CR) 10Hashar: "Should be a noop" [integration/config] - 10https://gerrit.wikimedia.org/r/224821 (owner: 10Hashar) [16:18:13] (03CR) 10Paladox: "Not sure weather to add check for jshint and jslint for users that are not added to the whitelist. npm checks them but doesent allow users" [integration/config] - 10https://gerrit.wikimedia.org/r/224819 (owner: 10Paladox) [16:20:20] RECOVERY - Free space - all mounts on deployment-bastion is OK All targets OK [16:26:44] (03PS1) 10Paladox: Add jshint to vector skin [integration/config] - 10https://gerrit.wikimedia.org/r/224824 [16:26:58] (03PS2) 10Paladox: Add jshint to vector skin [integration/config] - 10https://gerrit.wikimedia.org/r/224824 [16:46:39] (03PS2) 10JanZerebecki: tests: increase nosetests verbosity [integration/config] - 10https://gerrit.wikimedia.org/r/224797 (owner: 10Hashar) [16:46:48] (03CR) 10JanZerebecki: [C: 032] tests: increase nosetests verbosity [integration/config] - 10https://gerrit.wikimedia.org/r/224797 (owner: 10Hashar) [16:48:46] (03Merged) 10jenkins-bot: tests: increase nosetests verbosity [integration/config] - 10https://gerrit.wikimedia.org/r/224797 (owner: 10Hashar) [17:09:33] (03PS3) 10Paladox: Add jshint to vector skin [integration/config] - 10https://gerrit.wikimedia.org/r/224824 [17:09:48] (03PS3) 10Paladox: Use npm for CollapsibleVector [integration/config] - 10https://gerrit.wikimedia.org/r/224819 [17:10:28] (03PS14) 10Paladox: Add jsonlint check for MaintenanceShell [integration/config] - 10https://gerrit.wikimedia.org/r/222592 [17:23:06] (03PS4) 10Paladox: Use npm for CollapsibleVector [integration/config] - 10https://gerrit.wikimedia.org/r/224819 [17:24:14] Yippee, build fixed! [17:24:14] Project beta-mediawiki-config-update-eqiad build #2908: FIXED in 2.1 sec: https://integration.wikimedia.org/ci/job/beta-mediawiki-config-update-eqiad/2908/ [17:24:40] (03PS4) 10Paladox: Add jshint to vector skin [integration/config] - 10https://gerrit.wikimedia.org/r/224824 [17:26:50] (03PS1) 10Paladox: Add jshint test for metrolook [integration/config] - 10https://gerrit.wikimedia.org/r/224834 [17:27:15] (03PS2) 10Paladox: Add jshint test for metrolook [integration/config] - 10https://gerrit.wikimedia.org/r/224834 [17:29:13] (03PS5) 10Paladox: Use npm for CollapsibleVector [integration/config] - 10https://gerrit.wikimedia.org/r/224819 [17:45:27] (03PS1) 10Paladox: Update theme tests [integration/config] - 10https://gerrit.wikimedia.org/r/224838 [17:45:35] (03PS2) 10Paladox: Update theme tests [integration/config] - 10https://gerrit.wikimedia.org/r/224838 [17:46:06] (03CR) 10Paladox: [C: 04-1] "Needs patch in the theme extension for this too work so will do it now." [integration/config] - 10https://gerrit.wikimedia.org/r/224838 (owner: 10Paladox) [17:53:36] !log cherry-picked https://gerrit.wikimedia.org/r/#/c/224829/ [17:53:40] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [17:57:29] marxarelli: Dónde estás? [17:58:03] Aqui [18:00:56] alla [18:06:49] marxarel_: btw, I think your nick being your non-standard one is what is causing you not to be able to join -team [18:07:14] too many channels [18:07:47] PROBLEM - Free space - all mounts on deployment-fluorine is CRITICAL deployment-prep.deployment-fluorine.diskspace.root.byte_percentfree (<100.00%) [18:07:50] super secret cabal channels, even [18:08:01] stupid free space issues [18:08:19] oh yeah, i'm using irccloud and limechat concurrently [18:26:50] (03PS1) 10Zfilipin: Run RuboCop experimentally for operations/puppet [integration/config] - 10https://gerrit.wikimedia.org/r/224850 (https://phabricator.wikimedia.org/T102020) [18:30:01] greg-g wants me to say that I'm breaking beta wiki, but really it's not so bad [18:30:06] I think the wifi is most of the problem [18:30:56] I've had two or three Wikimedia Errors probably due to timeouts or something [18:31:27] (03CR) 10Hashar: [C: 032] "Go go go!" [integration/config] - 10https://gerrit.wikimedia.org/r/224850 (https://phabricator.wikimedia.org/T102020) (owner: 10Zfilipin) [18:33:26] (03Merged) 10jenkins-bot: Run RuboCop experimentally for operations/puppet [integration/config] - 10https://gerrit.wikimedia.org/r/224850 (https://phabricator.wikimedia.org/T102020) (owner: 10Zfilipin) [18:34:44] (03CR) 10Hashar: "Ran 'check experimental' on the rubocop patch at https://gerrit.wikimedia.org/r/#/c/218389/" [integration/config] - 10https://gerrit.wikimedia.org/r/224850 (https://phabricator.wikimedia.org/T102020) (owner: 10Zfilipin) [18:36:07] 10Continuous-Integration-Infrastructure, 6operations: Create a basic RSpec unit test for operations/puppet - https://phabricator.wikimedia.org/T78342#1454759 (10zeljkofilipin) [18:52:13] (03PS3) 10Hashar: Add jshint test for metrolook [integration/config] - 10https://gerrit.wikimedia.org/r/224834 (owner: 10Paladox) [18:56:11] (03CR) 10Hashar: [C: 04-1] Add jshint test for metrolook (032 comments) [integration/config] - 10https://gerrit.wikimedia.org/r/224834 (owner: 10Paladox) [18:58:17] (03PS15) 10Hashar: Add jsonlint check for MaintenanceShell [integration/config] - 10https://gerrit.wikimedia.org/r/222592 (owner: 10Paladox) [18:58:34] (03CR) 10Hashar: [C: 032] Add jsonlint check for MaintenanceShell [integration/config] - 10https://gerrit.wikimedia.org/r/222592 (owner: 10Paladox) [19:00:38] (03Merged) 10jenkins-bot: Add jsonlint check for MaintenanceShell [integration/config] - 10https://gerrit.wikimedia.org/r/222592 (owner: 10Paladox) [19:06:33] (03CR) 10Hashar: [C: 04-1] "Since that repo has npm, the jshint job is better replaced by having users to run 'npm test' locally before sending patchsets." [integration/config] - 10https://gerrit.wikimedia.org/r/224824 (owner: 10Paladox) [19:10:59] (03PS4) 10Paladox: Add jshint test for metrolook [integration/config] - 10https://gerrit.wikimedia.org/r/224834 [19:12:02] (03PS5) 10Paladox: Add jshint test for metrolook [integration/config] - 10https://gerrit.wikimedia.org/r/224834 [19:13:04] (03CR) 10Paladox: "Thankyou." [integration/config] - 10https://gerrit.wikimedia.org/r/224834 (owner: 10Paladox) [19:16:20] (03CR) 10Paladox: "Hi maybe could we open npm to everyone since doing npm locally may be harder for some users that may be in experienced with it or just wan" [integration/config] - 10https://gerrit.wikimedia.org/r/224824 (owner: 10Paladox) [19:16:42] (03PS5) 10Paladox: Add jshint to vector skin [integration/config] - 10https://gerrit.wikimedia.org/r/224824 [19:17:29] (03PS6) 10Paladox: Add jshint to vector skin [integration/config] - 10https://gerrit.wikimedia.org/r/224824 [19:18:08] (03CR) 10Paladox: "Thanks." [integration/config] - 10https://gerrit.wikimedia.org/r/222592 (owner: 10Paladox) [19:19:38] (03PS6) 10Paladox: Use npm for CollapsibleVector [integration/config] - 10https://gerrit.wikimedia.org/r/224819 [19:20:23] (03PS7) 10Paladox: Use npm for CollapsibleVector [integration/config] - 10https://gerrit.wikimedia.org/r/224819 [19:21:45] marktraceur, greg-g, anyone know what's wrong with sync on Beta? [19:23:29] no [19:23:43] thcipriani: marxarelli ostriches ^ [19:23:56] could be the low disk space issue bd808 was looking into [19:26:35] greg-g, https://phabricator.wikimedia.org/P970 [19:27:46] mwdeploy is not allowed to run sudo on deployment-mediawiki02. This incident will be reported. [19:27:49] seems odd [19:28:00] Some i18ncache stuff was wasting a ton of space in /tmp. [19:28:02] I tried that because I got an error on eval.php about echowikis, and I saw the mediawiki-config sync was out of date. [19:28:02] On deployment-bastion, at least. [19:28:31] https://phabricator.wikimedia.org/P971 [19:28:35] ^ Original error I hit. [19:28:46] greg-g: That might be something I did [19:29:04] the perms thing [19:29:41] echowikis is not in mediawiki-config at all anymore, so the sync is out of date. [19:30:10] * bd808 will be looking at beta cluster scap [19:46:25] (03PS3) 10Paladox: Update theme tests [integration/config] - 10https://gerrit.wikimedia.org/r/224838 [19:53:47] sync should start working again here shortly, sudoers rule problem [19:59:25] thcipriani is my hero. I was ranting and he was fixing [20:01:08] typical [20:02:20] greg-g: I'm a manager now. It's what I do [20:03:04] bd808: :) [20:09:06] (03PS4) 10Paladox: Update theme tests [integration/config] - 10https://gerrit.wikimedia.org/r/224838 [20:11:04] (03PS5) 10Paladox: Update theme tests [integration/config] - 10https://gerrit.wikimedia.org/r/224838 [20:40:26] (03PS1) 10Dduvall: Expose current Environment instance for `page_url` interpolation [selenium] - 10https://gerrit.wikimedia.org/r/224949 [20:48:03] zeljkof-conferen: ^ [20:53:14] !log Added JanZerebecki as deployment-prep root [20:53:17] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [20:54:42] (03CR) 10Zfilipin: [C: 032] Expose current Environment instance for `page_url` interpolation [selenium] - 10https://gerrit.wikimedia.org/r/224949 (owner: 10Dduvall) [20:55:55] (03Merged) 10jenkins-bot: Expose current Environment instance for `page_url` interpolation [selenium] - 10https://gerrit.wikimedia.org/r/224949 (owner: 10Dduvall) [21:18:25] PROBLEM - Puppet failure on deployment-parsoidcache02 is CRITICAL 100.00% of data above the critical threshold [0.0] [21:26:16] PROBLEM - Puppet failure on deployment-cache-text03 is CRITICAL 100.00% of data above the critical threshold [0.0] [21:37:42] https://integration.wikimedia.org/ci/view/Beta/job/beta-scap-eqiad/61568/console [21:37:45] Beta scap failing [21:43:19] Parse error: syntax error, unexpected $end, expecting ')' in /srv/mediawiki-staging/wmf-config/ExtensionMessages-master.php on line 108 [21:53:05] 6Release-Engineering, 6Team-Practices: Make log responsibilities changes - https://phabricator.wikimedia.org/T89049#1455423 (10MBinder_WMF) @greg @hashar Anything needed from TPG for this? If not, we'd like to remove it from our board, assuming it remains on yours. :) [21:55:10] thcipriani: Is that related to my issue? [21:55:43] marktraceur: yeah, looking to see why that file isn't generating fully during scap to beta [21:56:40] K. Thanks! [21:57:37] 6Release-Engineering, 6Team-Practices: Make log responsibilities changes - https://phabricator.wikimedia.org/T89049#1455427 (10greg) 5Open>3Resolved I'm going to call this done, actually. @demon and company have done awesome things lately and we're at a MUCH better place than we were 4 months ago. [22:18:19] (03PS1) 10Dduvall: WIP [selenium] - 10https://gerrit.wikimedia.org/r/224964 [22:21:15] Yippee, build fixed! [22:21:15] Project beta-update-databases-eqiad build #1495: FIXED in 1 min 13 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/1495/ [22:25:41] (03Abandoned) 10Dduvall: WIP [selenium] - 10https://gerrit.wikimedia.org/r/224964 (owner: 10Dduvall) [22:27:31] Yippee, build fixed! [22:27:32] Project beta-scap-eqiad build #61574: FIXED in 16 min: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/61574/ [23:20:10] PROBLEM - Puppet failure on deployment-bastion is CRITICAL 100.00% of data above the critical threshold [0.0] [23:25:15] 10Beta-Cluster, 10Continuous-Integration-Infrastructure, 10pywikibot-core, 5Patch-For-Review: Run pywikibot test suite regularly on beta cluster as part of MediaWiki/Wikimedia CI - https://phabricator.wikimedia.org/T100903#1455754 (10Krenair) [23:25:17] 10Beta-Cluster, 5Patch-For-Review: Beta should not use productions interwiki.cdb - https://phabricator.wikimedia.org/T69931#1455752 (10Krenair) 5Open>3Resolved a:3Krenair