[00:01:15] 3Beta-Cluster, monitoring: Monitor Varnish caches on beta cluster have two varnishd process running - https://phabricator.wikimedia.org/T75944#786646 (10hashar) [00:02:01] 3Beta-Cluster: upload.beta.wmflabs.org is throwing 503s - https://phabricator.wikimedia.org/T75922#785763 (10hashar) And filled another task to add a monitoring probe on the beta cluster to verify two varnishd are running: {T75944} [00:02:31] so [00:02:33] should I sleep [00:02:44] or try to follow Greg Phabricator foot steps? [00:03:04] chrismcmahon: looks like elasticsearch released a paper on testing. They do a pretty thorough job. Compared to Mediawiki core they are angels. They don't have nearly our plugin problem or release pace though. Link: http://www.elasticsearch.org/content/uploads/2014/11/Elasticsearch-Whitepaper-Testing-Automation-for-Distributed-Applications.pdf [00:03:08] hashar: sleep now [00:03:23] there is just that one little task I can probably fix in a couple minutes [00:03:28] 3Beta-Cluster: upload.beta.wmflabs.org is throwing 503s - https://phabricator.wikimedia.org/T75922#786663 (10greg) [00:05:42] (03PS2) 10Dduvall: split function to reduce cyclomatic complexity (per rubocop) [ruby/api] - 10https://gerrit.wikimedia.org/r/175686 (owner: 10Stan) [00:06:22] (03CR) 10Dduvall: [C: 032] split function to reduce cyclomatic complexity (per rubocop) [ruby/api] - 10https://gerrit.wikimedia.org/r/175686 (owner: 10Stan) [00:06:35] Anybody know how or why all of the cherry picks on deployment-salt got nuked? [00:06:36] (03Merged) 10jenkins-bot: split function to reduce cyclomatic complexity (per rubocop) [ruby/api] - 10https://gerrit.wikimedia.org/r/175686 (owner: 10Stan) [00:06:36] 3Beta-Cluster, monitoring: Monitor Varnish caches on beta cluster have two varnishd process running - https://phabricator.wikimedia.org/T75944#786681 (10greg) [00:06:42] (03PS3) 10Dduvall: module/class doc comments (per rubocop) [ruby/api] - 10https://gerrit.wikimedia.org/r/175687 (owner: 10Stan) [00:06:51] manybubbles: neato. that's a lot of monkeys :-) [00:07:18] chrismcmahon: yeah! their robot butler is almost as busy as ours. Mostly because they test on so many platforms [00:08:35] 3Beta-Cluster: upload.beta.wmflabs.org is throwing 503s - https://phabricator.wikimedia.org/T75922#786688 (10greg) p:5Volunteer?>3Unbreak! [00:08:38] I like the permutations [00:08:42] (03CR) 10Dduvall: [C: 032] module/class doc comments (per rubocop) [ruby/api] - 10https://gerrit.wikimedia.org/r/175687 (owner: 10Stan) [00:08:59] (Sometimes phab tries to help you too much by changing the priority of tasks when you drag them to a new column) [00:09:20] bd808: no, and you might be missing hashar's last minutes [00:09:54] what? [00:10:07] (03PS2) 10Dduvall: use standard regex syntax (per rubocop) [ruby/api] - 10https://gerrit.wikimedia.org/r/175688 (owner: 10Stan) [00:10:09] oh I am not dieing [00:10:17] haha [00:10:18] but will sleep eventually soonish [00:10:22] well that's good [00:10:22] suspend'ing [00:10:59] So the only cherry-picks on deployment-salt are some version of my local hack patches [00:11:12] bd808: hate to say it, but can you file a bug? [00:11:27] I know that we had at least 5 more patches that have vanished [00:11:39] :/ [00:11:49] greg-g: I can but probably only after I stop cursing and put all my hacks back [00:11:59] :( :( :( [00:12:08] * bd808 blames obama [00:12:10] bd808: did you have them anywhere else? [00:12:28] They are all cherry-picks from gerrit, but I have to track them down [00:12:29] are they just sitting in gerrit needing to be cherry-picked into place? [00:12:31] * greg-g nods [00:13:04] (03PS3) 10Dduvall: use standard regex syntax (per rubocop) [ruby/api] - 10https://gerrit.wikimedia.org/r/175688 (owner: 10Stan) [00:13:30] 3Beta-Cluster: Make deployment prep have continuous replication lag - https://phabricator.wikimedia.org/T59583#786699 (10greg) [00:14:55] maybe there aren't too many missing. I've confirmed 2 so far [00:15:00] bd808: could we pass the (change,patchset) with Hiera: ? [00:15:11] so cherry picking would be a change to [[Hiera:deployment-prep]] :-D [00:15:26] hashar: That would be cute. [00:15:35] and theoretically possible [00:15:39] but a lot of work [00:15:41] then whatever script that needs to be written will stop puppet master, reset hard, iterate the list of patches, cherry pick them, start puppet [00:15:57] OR [00:16:01] puppet environnements :-] [00:16:22] I see 3 stashes. Maybe one of them has the good stuff in it [00:16:35] git reflag most probably has [00:16:56] reflog is a mess because of the hourly rebase [00:17:06] but yeah I'm sure they are in there somewhere [00:17:16] I have some hashes from the logs [00:24:55] (03PS1) 10Dduvall: Enable rspec checks for mediawiki-ruby-api [integration/config] - 10https://gerrit.wikimedia.org/r/175900 [00:25:39] (03CR) 10Dduvall: [C: 032] use standard regex syntax (per rubocop) [ruby/api] - 10https://gerrit.wikimedia.org/r/175688 (owner: 10Stan) [00:26:05] hashar: Thanks! [00:26:17] hashar: (Re. beta labs 503s) /cc ryasmeen. [00:27:27] James_F: beers welcomed ;] [00:27:49] hashar: See you in January. ;-) [00:28:57] hashar drinks for free in SF :-) [00:29:08] 3Beta-Cluster: Local cherry-picks missing on deployment-salt - https://phabricator.wikimedia.org/T75947#786742 (10bd808) [00:32:39] 3Beta-Cluster: Local cherry-picks missing on deployment-salt - https://phabricator.wikimedia.org/T75947#786762 (10bd808) Looking at `git reflog` this may have be caused by someone fixing a merge conflict between one of the `[LOCAL HACK]` patches and the upstream. I think these are the only patches that were drop... [00:36:24] 3Beta-Cluster: Local cherry-picks missing on deployment-salt - https://phabricator.wikimedia.org/T75947#786774 (10bd808) Reapplied via local cherry-pick (eg `git cherry-pick ac13d48`). Here's the local patch stack now: ``` $ git log --color --pretty=oneline --abbrev-commit origin/HEAD..HEAD 59ab71d logstash: Ru... [00:38:28] (03PS2) 10Dduvall: remove rubocop todo [ruby/api] - 10https://gerrit.wikimedia.org/r/175689 (owner: 10Stan) [00:39:29] (03CR) 10Dduvall: [C: 032] "Thanks!" [ruby/api] - 10https://gerrit.wikimedia.org/r/175689 (owner: 10Stan) [00:39:49] (03PS2) 10Dduvall: correct cop name [ruby/api] - 10https://gerrit.wikimedia.org/r/175690 (owner: 10Stan) [00:40:57] (03CR) 10Dduvall: [C: 032] "Thanks!" [ruby/api] - 10https://gerrit.wikimedia.org/r/175690 (owner: 10Stan) [00:48:15] 3Beta-Cluster: Local cherry-picks missing on deployment-salt - https://phabricator.wikimedia.org/T75947#786812 (10bd808) [00:48:39] greg-g: https://phabricator.wikimedia.org/T75947 -- I think I've got things back to normal. [00:49:55] bd808: thanks man, and sorry :/ [00:50:51] !log Restored puppet cherry-picks from reflog [phab:T75947] [00:50:53] Logged the message, Master [00:51:48] !log cherry-picked patch for redis logstash input from MW {{gerrit|175896}} [00:51:49] Logged the message, Master [00:52:09] greg-g: It was like old home week :) [00:57:44] 3Beta-Cluster: Local cherry-picks missing on deployment-salt - https://phabricator.wikimedia.org/T75947#786845 (10hashar) The [LOCAL HACK] patches definitely need to be sent to Gerrit if not already.. There is a task filled for one ( T67591 ), the `Change MySQL admin user in sql script` could use task/gerrit ch... [00:57:58] opwell [00:58:03] farewell [00:58:06] hast la vista [00:58:14] whatever. I am really sleeping now ;] [01:03:44] hashar: see ya! [01:27:41] 3Beta-Cluster, MediaWiki-extensions-GettingStarted: GettingStarted extension broken at beta.wmflabs.org - https://phabricator.wikimedia.org/T53362#786995 (10Mattflaschen) The 'gettingstarted' tour (that exact name, there are still two other GettingStarted tours, 'gettingstartedtasktoolbar' and 'gettingstartedtas... [01:27:46] Yippee, build fixed! [01:27:46] Project browsertests-Flow-test2.wikipedia.org-linux-chrome-sauce build #295: FIXED in 1 hr 0 min: https://integration.wikimedia.org/ci/job/browsertests-Flow-test2.wikipedia.org-linux-chrome-sauce/295/ [02:48:30] Project browsertests-UniversalLanguageSelector-commons.wikimedia.beta.wmflabs.org-linux-firefox-sauce build #302: FAILURE in 31 min: https://integration.wikimedia.org/ci/job/browsertests-UniversalLanguageSelector-commons.wikimedia.beta.wmflabs.org-linux-firefox-sauce/302/ [03:35:12] Project browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_7-internet_explorer-11-sauce build #98: FAILURE in 33 min: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_7-internet_explorer-11-sauce/98/ [04:30:48] Yippee, build fixed! [04:30:49] Project browsertests-UploadWizard-commons.wikimedia.beta.wmflabs.org-linux-chrome-sauce build #318: FIXED in 10 min: https://integration.wikimedia.org/ci/job/browsertests-UploadWizard-commons.wikimedia.beta.wmflabs.org-linux-chrome-sauce/318/ [04:54:28] Yippee, build fixed! [04:54:28] Project browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #378: FIXED in 1 hr 4 min: https://integration.wikimedia.org/ci/job/browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/378/ [04:55:12] Project beta-scap-eqiad build #31315: FAILURE in 1 min 8 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/31315/ [05:15:31] Yippee, build fixed! [05:15:31] Project beta-scap-eqiad build #31317: FIXED in 1 min 14 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/31317/ [05:44:03] (03PS1) 10Krinkle: mediawiki-core-qunit: Add archive-log-dir publisher [integration/config] - 10https://gerrit.wikimedia.org/r/175947 [05:49:51] (03CR) 10Krinkle: [C: 032] mediawiki-core-qunit: Add archive-log-dir publisher [integration/config] - 10https://gerrit.wikimedia.org/r/175947 (owner: 10Krinkle) [05:53:33] (03Merged) 10jenkins-bot: mediawiki-core-qunit: Add archive-log-dir publisher [integration/config] - 10https://gerrit.wikimedia.org/r/175947 (owner: 10Krinkle) [06:00:21] Yippee, build fixed! [06:00:22] Project browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-firefox-monobook-sauce build #152: FIXED in 1 hr 5 min: https://integration.wikimedia.org/ci/job/browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-firefox-monobook-sauce/152/ [06:02:32] (03PS3) 10Krinkle: mwconf: Remove references to pmtpa [integration/jenkins] - 10https://gerrit.wikimedia.org/r/173518 [06:04:46] (03CR) 10Krinkle: [C: 032] mwconf: Remove references to pmtpa [integration/jenkins] - 10https://gerrit.wikimedia.org/r/173518 (owner: 10Krinkle) [06:04:50] (03Merged) 10jenkins-bot: mwconf: Remove references to pmtpa [integration/jenkins] - 10https://gerrit.wikimedia.org/r/173518 (owner: 10Krinkle) [06:05:11] (03Abandoned) 10Krinkle: [WIP] Experimental mediawiki-core-qunit-labsSlave [integration/config] - 10https://gerrit.wikimedia.org/r/174731 (owner: 10Krinkle) [06:27:22] Yippee, build fixed! [06:27:23] Project browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce build #346: FIXED in 57 min: https://integration.wikimedia.org/ci/job/browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce/346/ [06:49:32] Project browsertests-CirrusSearch-test2.wikipedia.org-linux-firefox-sauce build #284: FAILURE in 2 min 41 sec: https://integration.wikimedia.org/ci/job/browsertests-CirrusSearch-test2.wikipedia.org-linux-firefox-sauce/284/ [07:01:17] Yippee, build fixed! [07:01:17] Project browsertests-VisualEditor-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #431: FIXED in 1 hr 15 min: https://integration.wikimedia.org/ci/job/browsertests-VisualEditor-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/431/ [07:48:46] 3Beta-Cluster, Wikimedia-Labs-Infrastructure: Setup real ssl certs for Beta Cluster using a restricted project - https://phabricator.wikimedia.org/T75919#787392 (10yuvipanda) Consider three things we want beta to have: # Same as prod (including SSL Certs) # sudo for as many people as possible They're incompati... [08:20:35] Yippee, build fixed! [08:20:35] Project browsertests-MobileFrontend-en.m.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #388: FIXED in 1 hr 27 min: https://integration.wikimedia.org/ci/job/browsertests-MobileFrontend-en.m.wikipedia.beta.wmflabs.org-linux-firefox-sauce/388/ [08:39:12] 3Beta-Cluster: API error: unable to connect to redis - https://phabricator.wikimedia.org/T74982#787512 (10Petrb) [09:05:00] Yippee, build fixed! [09:05:01] Project browsertests-Flow-en.wikipedia.beta.wmflabs.org-windows_8-internet_explorer-sauce build #297: FIXED in 44 min: https://integration.wikimedia.org/ci/job/browsertests-Flow-en.wikipedia.beta.wmflabs.org-windows_8-internet_explorer-sauce/297/ [09:29:26] Project browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_8-internet_explorer-sauce build #307: FAILURE in 32 min: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_8-internet_explorer-sauce/307/ [09:41:25] (03PS2) 10Zfilipin: Enable rspec checks for mediawiki-ruby-api [integration/config] - 10https://gerrit.wikimedia.org/r/175900 (owner: 10Dduvall) [09:50:51] 3Beta-Cluster: Setup monitoring for Beta cluster - https://phabricator.wikimedia.org/T53497#787642 (10zeljkofilipin) [09:51:33] 3Release-Engineering: Can "gem build" fail if there are _any_ warnings? - https://phabricator.wikimedia.org/T1333#787643 (10zeljkofilipin) Will ask upstream, thanks hashar. [09:51:54] 3Release-Engineering: Can "gem build" fail if there are _any_ warnings? - https://phabricator.wikimedia.org/T1333#787644 (10zeljkofilipin) a:3zeljkofilipin [09:59:05] 3Beta-Cluster: beta labs no longer listens for HTTPS - https://phabricator.wikimedia.org/T70387#787645 (10zeljkofilipin) [09:59:36] 3Beta-Cluster: sync Sandbox gadget from production to en.wikipedia.beta.wmflabs.org - https://phabricator.wikimedia.org/T49205#787646 (10zeljkofilipin) [10:00:17] 3Beta-Cluster, Release-Engineering: Use Beta cluster as a true canary for code deployments (tracking) - https://phabricator.wikimedia.org/T53494#787647 (10zeljkofilipin) [10:00:43] 3Scrum-of-Scrums, Beta-Cluster: beta cluster: deployment-cache-upload02 does not seem to purge content when getting PURGE - https://phabricator.wikimedia.org/T67683#787648 (10zeljkofilipin) [10:28:44] 3Release-Engineering, Continuous-Integration: doc.wikimedia.org: Generate documentation for release tags (tracking) - https://phabricator.wikimedia.org/T73062#787680 (10hashar) [10:29:11] 3Release-Engineering, Continuous-Integration: doc.wikimedia.org: Generate documentation for release tags (tracking) - https://phabricator.wikimedia.org/T73062#771415 (10hashar) [10:29:29] 3Release-Engineering, Continuous-Integration: doc.wikimedia.org: Generate documentation for release tags (tracking) - https://phabricator.wikimedia.org/T73062#771415 (10hashar) [10:32:54] zeljkof: per your comment on https://phabricator.wikimedia.org/T72605 can I assume it's ok to work on https://phabricator.wikimedia.org/T75898? [10:37:05] stan: go ahead! :) [10:50:33] Project browsertests-Flow-test2.wikipedia.org-linux-chrome-sauce build #296: FAILURE in 1 hr 1 min: https://integration.wikimedia.org/ci/job/browsertests-Flow-test2.wikipedia.org-linux-chrome-sauce/296/ [11:15:25] Project beta-scap-eqiad build #31352: FAILURE in 1 min 22 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/31352/ [11:35:30] Yippee, build fixed! [11:35:31] Project beta-scap-eqiad build #31354: FIXED in 1 min 26 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/31354/ [11:47:19] (03PS3) 10Hashar: Generate mw/core doxygen on new release [integration/config] - 10https://gerrit.wikimedia.org/r/175712 [11:47:21] (03PS1) 10Hashar: Soften branch filter for publish jobs [integration/config] - 10https://gerrit.wikimedia.org/r/175977 [11:50:52] (03CR) 10Hashar: "Rebased on top of https://gerrit.wikimedia.org/r/#/c/175977/ which adds to all publish jobs:" [integration/config] - 10https://gerrit.wikimedia.org/r/175712 (owner: 10Hashar) [11:52:40] (03Abandoned) 10Hashar: Soften branch filter for publish jobs [integration/config] - 10https://gerrit.wikimedia.org/r/175977 (owner: 10Hashar) [11:53:35] (03PS4) 10Hashar: Generate mw/core doxygen on new release [integration/config] - 10https://gerrit.wikimedia.org/r/175712 [11:54:45] (03CR) 10Hashar: [C: 032] "Lets try!" [integration/config] - 10https://gerrit.wikimedia.org/r/175712 (owner: 10Hashar) [11:56:13] (03Merged) 10jenkins-bot: Generate mw/core doxygen on new release [integration/config] - 10https://gerrit.wikimedia.org/r/175712 (owner: 10Hashar) [15:16:36] 3Release-Engineering: Can "gem build" fail if there are _any_ warnings? - https://phabricator.wikimedia.org/T1333#788171 (10hashar) @zeljkofilipin The bash script above can be integrated right now in our '{name}-gembuild' job template ( in jjb/mediawiki-misc.yaml) :) [15:23:33] 3Beta-Cluster, Wikimedia-Labs-Infrastructure: Setup real ssl certs for Beta Cluster using a restricted project - https://phabricator.wikimedia.org/T75919#788195 (10hashar) >>! In T75919#785867, @yuvipanda wrote: > Can't actually use dynamicproxy, since the cert there is just for *.wmflabs.org. Also, this termina... [15:26:10] 3Beta-Cluster, Wikimedia-Labs-Infrastructure: Setup real ssl certs for Beta Cluster using a restricted project - https://phabricator.wikimedia.org/T75919#788206 (10yuvipanda) I'll also note that from a security perspective, since people without NDA *had* root in the past, it would've been trivial for them to acq... [15:31:34] 3Beta-Cluster, Wikimedia-Labs-Infrastructure: Setup real ssl certs for Beta Cluster using a restricted project - https://phabricator.wikimedia.org/T75919#788221 (10hashar) >>! In T75919#786302, @greg wrote: >>>! In T75919#786292, @hashar wrote: >> I loose track on the rather long {T50501} , but I commented about... [15:53:51] 3Beta-Cluster, Wikimedia-Labs-Infrastructure: Setup real ssl certs for Beta Cluster using a restricted project - https://phabricator.wikimedia.org/T75919#788261 (10greg) >>! In T75919#788221, @hashar wrote: >>>! In T75919#786302, @greg wrote: >>>>! In T75919#786292, @hashar wrote: >>> I loose track on the rather... [16:45:27] 3Beta-Cluster, Wikimedia-Labs-Infrastructure: Setup real ssl certs for Beta Cluster using a restricted project - https://phabricator.wikimedia.org/T75919#788525 (10chasemp) >>! In T75919#788221, @hashar wrote: >>>! In T75919#786302, @greg wrote: >>>>! In T75919#786292, @hashar wrote: >>> I loose track on the rat... [16:57:25] 3Beta-Cluster, Wikimedia-Labs-Infrastructure: Setup real ssl certs for Beta Cluster using a restricted project - https://phabricator.wikimedia.org/T75919#788615 (10bd808) Self-signed SSL certificate chains have historically been a huge pain for automated browser testing. @Cmcmahon has pointed this out elsewhere... [16:59:20] 3Beta-Cluster, Wikimedia-Labs-Infrastructure: Setup real ssl certs for Beta Cluster using a restricted project - https://phabricator.wikimedia.org/T75919#788637 (10greg) [17:10:44] 3MediaWiki-JobRunner, Beta-Cluster, MediaWiki-Core-Team: beta cluster job runner keep running some periodic tasks - https://phabricator.wikimedia.org/T65681#788656 (10greg) [17:14:37] Yippee, build fixed! [17:14:37] Project browsertests-Wikidata-WikidataTests-linux-firefox-sauce build #54: FIXED in 2 hr 52 min: https://integration.wikimedia.org/ci/job/browsertests-Wikidata-WikidataTests-linux-firefox-sauce/54/ [17:15:37] 3MediaWiki-JobRunner, Beta-Cluster, Scrum-of-Scrums, MediaWiki-Core-Team: beta cluster job runner keep running some periodic tasks - https://phabricator.wikimedia.org/T65681#788663 (10greg) [17:35:19] Project beta-scap-eqiad build #31387: FAILURE in 1 min 18 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/31387/ [17:55:29] Yippee, build fixed! [17:55:30] Project beta-scap-eqiad build #31389: FIXED in 1 min 21 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/31389/ [18:02:21] 3Release-Engineering: Mukunda ready to do deploy on 12/16 - https://phabricator.wikimedia.org/T76049 (10greg) 3NEW p:3High [18:05:35] 3Release-Engineering: Mukunda ready to do deploy on 12/16 - https://phabricator.wikimedia.org/T76049#788768 (10greg) [18:37:33] 3Scrum-of-Scrums, Beta-Cluster: File upload area resorts to 0777 permissions to for uploaded content - https://phabricator.wikimedia.org/T75206#788836 (10Milimetric) [18:37:39] 3Scrum-of-Scrums, Beta-Cluster: beta cluster: deployment-cache-upload02 does not seem to purge content when getting PURGE - https://phabricator.wikimedia.org/T67683#788838 (10Milimetric) 5Open>3Resolved [19:03:57] 3Scrum-of-Scrums, Beta-Cluster: beta cluster: deployment-cache-upload02 does not seem to purge content when getting PURGE - https://phabricator.wikimedia.org/T67683#788905 (10greg) [19:18:50] Is Jenkins off being an asshole? [19:18:58] https://integration.wikimedia.org/ci/job/mwext-UploadWizard-npm/259/console 404's [19:22:37] 3Release-Engineering: l10nupdate broken in production with scap related errors - https://phabricator.wikimedia.org/T1383#789022 (10bd808) Fix was https://gerrit.wikimedia.org/r/#/c/174784/ [19:40:19] 3Scrum-of-Scrums, Beta-Cluster: File upload area resorts to 0777 permissions to for uploaded content - https://phabricator.wikimedia.org/T75206#789051 (10greg) [20:39:38] (03PS1) 10Hashar: wmf/* tags no more trigger mediawiki-core-release [integration/config] - 10https://gerrit.wikimedia.org/r/176053 [20:45:43] 3Scrum-of-Scrums, Beta-Cluster: beta cluster: deployment-cache-upload02 does not seem to purge content when getting PURGE - https://phabricator.wikimedia.org/T67683#789248 (10greg) Reopening as nothing was done here. (cc @cmcmahon @milimetric) [20:46:33] 3Scrum-of-Scrums, Beta-Cluster: Create labs ldap user and group "apache" to be used for NFS4 access in deployment-prep project - https://phabricator.wikimedia.org/T76086#789254 (10bd808) [20:46:54] 3Beta-Cluster: File upload area resorts to 0777 permissions to for uploaded content - https://phabricator.wikimedia.org/T75206#789261 (10bd808) [20:47:46] 3Beta-Cluster: File upload area resorts to 0777 permissions to for uploaded content - https://phabricator.wikimedia.org/T75206#748930 (10bd808) [20:47:52] (03PS2) 10Hashar: wmf/* tags no more trigger mediawiki-core-release [integration/config] - 10https://gerrit.wikimedia.org/r/176053 [20:49:28] (03CR) 10Hashar: [C: 032] wmf/* tags no more trigger mediawiki-core-release [integration/config] - 10https://gerrit.wikimedia.org/r/176053 (owner: 10Hashar) [20:50:01] 3Scrum-of-Scrums, Beta-Cluster: Create labs ldap user and group "apache" to be used for NFS4 access in deployment-prep project - https://phabricator.wikimedia.org/T76086#789273 (10bd808) [20:50:22] (03Merged) 10jenkins-bot: wmf/* tags no more trigger mediawiki-core-release [integration/config] - 10https://gerrit.wikimedia.org/r/176053 (owner: 10Hashar) [21:09:13] 3Wikimedia-Logstash, Release-Engineering: Upgrade ElasticSearch on logstash boxen to 1.3.5 - https://phabricator.wikimedia.org/T76089 (10Reedy) 3NEW p:3Triage [21:13:50] hashar: if you're here, does zuul need to be kicked? a number of jobs seem to be queued, but zuul looks stuck [21:13:57] 3Wikimedia-Logstash, Release-Engineering: Icinga monitoring for elasticsearch doesn't notice OOM conditions - https://phabricator.wikimedia.org/T76090#789335 (10bd808) [21:14:18] 3Beta-Cluster: Delete /srv/vdb/varnish* files on all Varnish instances - https://phabricator.wikimedia.org/T76091 (10hashar) 3NEW p:3Triage [21:15:03] chrismcmahon: looking [21:17:24] chrismcmahon: fixed :] [21:17:59] 3Wikimedia-Logstash, Release-Engineering: Icinga monitoring for elasticsearch doesn't notice OOM conditions - https://phabricator.wikimedia.org/T76090#789365 (10bd808) I think I remember there being a java jre command line flag that lets you install an OOM signal handler. I have vague memories of using that to s... [21:18:04] thanks hashar [21:18:44] chrismcmahon: the workaround is https://phabricator.wikimedia.org/T65760#688180 :D [21:56:55] 3VisualEditor, Beta-Cluster: VisualEditor not updating in beta labs - https://phabricator.wikimedia.org/T76097#789475 (10Catrope) [22:14:52] 3VisualEditor, Beta-Cluster: VisualEditor not updating in beta labs - https://phabricator.wikimedia.org/T76097#789504 (10bd808) deployment-bastion:/srv/mediawiki-staging/php-master/extensions/VisualEditor is at git hash a14f88d which seems to match https://git.wikimedia.org/summary/mediawiki%2Fextensions%2FVisua... [22:23:42] Project beta-mediawiki-config-update-eqiad build #1502: FAILURE in 2.9 sec: https://integration.wikimedia.org/ci/job/beta-mediawiki-config-update-eqiad/1502/ [22:28:51] 3VisualEditor, Beta-Cluster: VisualEditor not updating in beta labs - https://phabricator.wikimedia.org/T76097#789542 (10bd808) [22:35:57] 3VisualEditor, Beta-Cluster: VisualEditor not updating in beta labs - https://phabricator.wikimedia.org/T76097#789575 (10Jdforrester-WMF) [22:43:22] zuul is stuck again. I'ma try hashar's workaround... [22:44:45] chrismcmahon: yeah I am on it :( [22:44:51] still https://phabricator.wikimedia.org/T72597 [22:45:00] hashar: just did the Gearman unset/set actions [22:45:04] it hasn't recovered yet, I need to full stop Jenkins / Zuul but need to take traces [22:45:04] I just did [22:45:15] 3VisualEditor, Beta-Cluster: VisualEditor not updating in beta labs - https://phabricator.wikimedia.org/T76097#789597 (10Jdforrester-WMF) [22:45:17] yeah that is more or less a different issue though [22:45:24] oh, I did not full stop Jenkins, no [22:45:29] the threads are in a deadlock and the workaround does not really handle it [22:48:23] 3Release-Engineering, Continuous-Integration: Jenkins Gearman plugin has deadlock on executor threads (was: Beta Cluster stopped receiving code updates (beta-update-databases-eqiad hung) - https://phabricator.wikimedia.org/T72597#789605 (10hashar) [22:48:43] 3Release-Engineering, Continuous-Integration: Jenkins Gearman plugin has deadlock on executor threads (was: Beta Cluster stopped receiving code updates (beta-update-databases-eqiad hung) - https://phabricator.wikimedia.org/T72597#747903 (10hashar) [22:49:09] 3Release-Engineering, Continuous-Integration: Jenkins Gearman plugin has deadlock on executor threads (was: Beta Cluster stopped receiving code updates (beta-update-databases-eqiad hung) - https://phabricator.wikimedia.org/T72597#747903 (10hashar) [23:38:26] 3VisualEditor, Beta-Cluster: [Regression pre-wmf10] upload.beta.wmflabs.org is throwing 503s so all images are appearing with a broken icon inside VE - https://phabricator.wikimedia.org/T75786#789747 (10Ryasmeen) [23:38:53] 3VisualEditor, Beta-Cluster: [Regression pre-wmf10] upload.beta.wmflabs.org is throwing 503s so Math function parsing is completely broken inside VE - https://phabricator.wikimedia.org/T75787#789748 (10Ryasmeen) [23:50:19] Project beta-scap-eqiad build #31422: FAILURE in 26 min: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/31422/