[00:11:34] 3Librarization, MediaWiki-Core-Team, Deployment-Systems: Have a check for reported security issues in dependencies - https://phabricator.wikimedia.org/T74193#983367 (10csteipp) >>! In T74193#980413, @hashar wrote: > Can we get the job defined with Jenkins Job Builder in `integration/config.git`. Multiple people... [00:27:55] (03PS1) 10Krinkle: macro: Remove unused 'wipe-workspace' builder [integration/config] - 10https://gerrit.wikimedia.org/r/185579 [00:27:57] (03PS1) 10Krinkle: macro: Factor out 'prepare-qunit-localhost' builder [integration/config] - 10https://gerrit.wikimedia.org/r/185580 [00:28:35] 3MediaWiki-extensions-UniversalLanguageSelector, MediaWiki-extensions-Babel, Continuous-Integration, MediaWiki-extensions-CLDR: Add Babel, cldr and UniversalLanguageSelector extensions to mediawiki-phpunit-(zend|hhvm) - https://phabricator.wikimedia.org/T86930#983401 (10Nemo_bis) > I would love to have all the e... [00:28:44] chrismcmahon: could i get a review from ya when you get a chance? [00:28:46] chrismcmahon: https://gerrit.wikimedia.org/r/#/c/185503/ [00:31:59] marxarelli: looking... [00:32:28] 3Continuous-Integration: Make sure everyone in RelEng has sudo on integration slaves in labs - https://phabricator.wikimedia.org/T86779#983404 (10Krinkle) If you define "everyone", we can add them via https://wikitech.wikimedia.org/wiki/Nova_Resource:Integration. Current admins: ``` # Labs User:novaadmin User:D... [00:35:06] marxarelli: you want that merged? I don't see why not... [00:38:53] chrismcmahon: yeah yeah yeah [00:38:56] :) [00:38:59] 3Librarization, MediaWiki-Core-Team, Deployment-Systems: Have a check for reported security issues in dependencies - https://phabricator.wikimedia.org/T74193#983416 (10Krinkle) Instead of adding the logic to a Jenkins job (whether via Jenkins web interface or via `integration/config.git`), I'd recommend adding t... [00:39:36] chrismcmahon: i have an announcement ready to roll as well [00:39:43] (03CR) 10Cmcmahon: [C: 032] "merging." [selenium] - 10https://gerrit.wikimedia.org/r/185503 (owner: 10Dduvall) [00:40:09] (03Merged) 10jenkins-bot: Releasing major version (in alpha) 1.0.0.pre.1 [selenium] - 10https://gerrit.wikimedia.org/r/185503 (owner: 10Dduvall) [00:40:11] chrismcmahon: danke! [00:40:12] marxarelli: good timing. I'm looking forward to using it [00:40:30] chrismcmahon: yeah, let's do some pairing next week [00:40:34] (03PS2) 10Krinkle: cleanup: Remove redundant job-group 'mediawiki-jobs' [integration/config] - 10https://gerrit.wikimedia.org/r/183507 [00:41:16] (03CR) 10Krinkle: [C: 032] macro: Remove unused 'wipe-workspace' builder [integration/config] - 10https://gerrit.wikimedia.org/r/185579 (owner: 10Krinkle) [00:42:00] chrismcmahon: so, the jenkins gembuild job doesn't publish the gem, right? [00:42:15] marxarelli: that, and I want to go back to my nested-elements spike for Flow with you and Zeljko also. Since they landed the TOC and removed collapse/expand it no longer rebases cleanly (which I figured) but I have some new stuff I want to try to make work [00:42:17] chrismcmahon: it just builds? [00:42:50] marxarelli: nope, you have to publish the gem by hand. I've never done it, it was always Zeljko who made that happen. [00:42:53] doesn't appear to... https://integration.wikimedia.org/ci/job/mediawiki-selenium-gembuild/399/console [00:43:15] chrismcmahon: cool. it's really easy (gem push) but i didn't want to jump the gun [00:43:42] chrismcmahon: yeah, let's pair on those flow refactorings as well [00:48:10] (03Merged) 10jenkins-bot: macro: Remove unused 'wipe-workspace' builder [integration/config] - 10https://gerrit.wikimedia.org/r/185579 (owner: 10Krinkle) [00:48:24] (03PS3) 10Krinkle: cleanup: Remove redundant job-group 'mediawiki-jobs' [integration/config] - 10https://gerrit.wikimedia.org/r/183507 [00:48:31] (03CR) 10Krinkle: [C: 032] cleanup: Remove redundant job-group 'mediawiki-jobs' [integration/config] - 10https://gerrit.wikimedia.org/r/183507 (owner: 10Krinkle) [00:49:02] (03PS2) 10Krinkle: macro: Factor out 'prepare-qunit-localhost' builder [integration/config] - 10https://gerrit.wikimedia.org/r/185580 [00:50:15] (03PS3) 10Krinkle: macro: Factor out 'prepare-qunit-localhost' builder [integration/config] - 10https://gerrit.wikimedia.org/r/185580 [00:56:09] (03Merged) 10jenkins-bot: cleanup: Remove redundant job-group 'mediawiki-jobs' [integration/config] - 10https://gerrit.wikimedia.org/r/183507 (owner: 10Krinkle) [00:59:59] 3Librarization, MediaWiki-Core-Team, Deployment-Systems: Have a check for reported security issues in dependencies - https://phabricator.wikimedia.org/T74193#983435 (10csteipp) [01:26:04] greg-g: I'd like to deploy https://gerrit.wikimedia.org/r/#/q/Ic1d10393912fcefa22d675fd4aa2baf437d2a05a,n,z today (!) to unbreak VE https://phabricator.wikimedia.org/T86270 [01:41:18] greg-g: Because of the breakage I'll go ahead and deploy it. It follows-up the deployment of https://gerrit.wikimedia.org/r/#/q/I335ad42bb83917b3bf36ad6ee1c45f4521377c80,n,z yesterday for EventLogging. [01:46:18] 3Phabricator: Cannot download original file from Phabricator - https://phabricator.wikimedia.org/T87083#983524 (10Tgr) [03:19:26] Can anyone remind me the magic incantation to get a stuck Beta Labs job to execute? https://integration.wikimedia.org/ci/view/Beta/ is showing it with the black clock which means it's delayed "unnecessarily", and deployment-bastion.eqiad is showing as six idle workers… [03:43:31] Yippee, build fixed! [03:43:31] Project browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-os_x_10.9-safari-sauce build #411: FIXED in 33 min: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-os_x_10.9-safari-sauce/411/ [03:46:44] Yippee, build fixed! [03:46:44] Project browsertests-CirrusSearch-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #395: FIXED in 1 min 26 sec: https://integration.wikimedia.org/ci/job/browsertests-CirrusSearch-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/395/ [03:52:42] PROBLEM - Puppet failure on deployment-parsoid05 is CRITICAL: CRITICAL: 11.11% of data above the critical threshold [0.0] [04:07:43] RECOVERY - Puppet failure on deployment-parsoid05 is OK: OK: Less than 1.00% above the threshold [0.0] [04:17:44] 3Beta-Cluster, Parsoid-Team: Cannot open VE in Betalabs , throwing error Error loading data from server: 503: parsoidserver-http: HTTP 503 - https://phabricator.wikimedia.org/T86951#983599 (10yuvipanda) I have temporarily reverted that change locally and things should work now. It looks like base::firewall is be... [04:35:49] Yippee, build fixed! [04:35:50] Project browsertests-Echo-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce build #292: FIXED in 8 min 23 sec: https://integration.wikimedia.org/ci/job/browsertests-Echo-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce/292/ [04:41:33] well [04:41:36] that’s just strange [04:41:49] where are you coming from, base::firewall? [04:41:50] whereeeee [05:23:27] Yippee, build fixed! [05:23:28] Project browsertests-VisualEditor-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce build #460: FIXED in 20 min: https://integration.wikimedia.org/ci/job/browsertests-VisualEditor-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce/460/ [05:26:48] 3Beta-Cluster, Parsoid-Team: Monitor the Parsoid backend service on beta cluster - https://phabricator.wikimedia.org/T87063#983617 (10yuvipanda) 5Open>3Resolved a:3yuvipanda Replicates the same check we have in prod, which just checks if port 8000 is responding.. http://shinken.wmflabs.org/service/deploym... [05:33:49] 3Beta-Cluster, Parsoid-Team: Cannot open VE in Betalabs , throwing error Error loading data from server: 503: parsoidserver-http: HTTP 503 - https://phabricator.wikimedia.org/T86951#983620 (10yuvipanda) 5Open>3Resolved After some more digging... https://gerrit.wikimedia.org/r/#/c/185428/ removed the ferm ru... [05:50:41] 3Beta-Cluster, Parsoid-Team: Monitor the Parsoid backend service on beta cluster - https://phabricator.wikimedia.org/T87063#983631 (10yuvipanda) [05:50:42] 3Beta-Cluster: Setup monitoring for Beta cluster (tracking) - https://phabricator.wikimedia.org/T53497#983630 (10yuvipanda) [05:51:23] 3Beta-Cluster: Monitor mathoid, cxserver, ciotid and apertium services on betacluster - https://phabricator.wikimedia.org/T87087#983632 (10yuvipanda) 3NEW [05:52:14] 3Beta-Cluster: Setup monitoring for Beta cluster (tracking) - https://phabricator.wikimedia.org/T53497#581090 (10yuvipanda) [05:52:15] 3Beta-Cluster: Monitor mathoid, cxserver, ciotid and apertium services on betacluster - https://phabricator.wikimedia.org/T87087#983643 (10yuvipanda) [06:00:37] 3Beta-Cluster: Monitor mathoid, cxserver, ciotid and apertium services on betacluster - https://phabricator.wikimedia.org/T87087#983649 (10yuvipanda) 5Open>3Resolved a:3yuvipanda Allll done [06:00:39] 3Beta-Cluster: Setup monitoring for Beta cluster (tracking) - https://phabricator.wikimedia.org/T53497#983652 (10yuvipanda) [06:30:40] bd808|BUFFER: if you’re in a pinch for logstash RAM, the lsearchd machines are lying around... [06:38:46] PROBLEM - Puppet failure on deployment-mediawiki03 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [06:43:11] Project browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce build #411: FAILURE in 24 min: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce/411/ [06:44:24] ^ is because I just restarted our DNS server [06:50:27] PROBLEM - Puppet failure on deployment-memc02 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [06:58:47] RECOVERY - Puppet failure on deployment-mediawiki03 is OK: OK: Less than 1.00% above the threshold [0.0] [07:15:24] RECOVERY - Puppet failure on deployment-memc02 is OK: OK: Less than 1.00% above the threshold [0.0] [07:57:12] 3Phabricator: Actions shouldn't be attributed to bzimport - https://phabricator.wikimedia.org/T847#983701 (10Nemo_bis) > We cannot time-travel to change the migration logic, proposing "declined" here again. Or this task could be kept open to figure out how to get everyone with actions to register. [08:47:41] PROBLEM - Puppet failure on deployment-bastion is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [09:02:23] RECOVERY - Puppet failure on deployment-stream is OK: OK: Less than 1.00% above the threshold [0.0] [09:02:38] RECOVERY - Puppet failure on deployment-bastion is OK: OK: Less than 1.00% above the threshold [0.0] [09:42:45] RECOVERY - Puppet failure on deployment-jobrunner01 is OK: OK: Less than 1.00% above the threshold [0.0] [09:43:14] 3Phabricator: Delete LikeLifer username - https://phabricator.wikimedia.org/T87092#983768 (10Qgil) 3NEW [10:06:36] greg-g: fixed a bunch more issues! puppet recoveries yay [10:09:33] 3Beta-Cluster: Setup monitoring for database servers in betalabs - https://phabricator.wikimedia.org/T87093#983793 (10yuvipanda) 3NEW a:3yuvipanda [10:12:05] (03CR) 10Hashar: "+2 for simplicity :-]" [integration/config] - 10https://gerrit.wikimedia.org/r/183507 (owner: 10Krinkle) [10:47:17] 3Phabricator: Phabricator far too spammy and off-topic to continue posting to wikibugs-l - https://phabricator.wikimedia.org/T763#983817 (10Bawolff) Fwiw i used to be subscribed a long time ago. It was a very good way to keep track of everything going on in our community, and helped me learn about how various th... [12:31:39] PROBLEM - Puppet failure on deployment-mx is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [13:16:49] 3Phabricator: T87062 contains a centralauth token and other interesting cookie information allowing to steal session - https://phabricator.wikimedia.org/T87100#983917 (10Rillke) 3NEW a:3Aklapper [13:17:26] 3Phabricator: T87062 contains a centralauth token and other interesting cookie information allowing to steal session - https://phabricator.wikimedia.org/T87100#983925 (10Rillke) Reference T87062. [14:02:09] 3Phabricator: "changed Security from none to none." - https://phabricator.wikimedia.org/T479#983947 (10Ciencia_Al_Poder) The "USERNAME set Security to none" action is still happening for tasks imported from bugzilla, and is confusing some users. See T20871#983927 >>! In T479#840405, @MZMcBride wrote: > As an in... [16:39:20] (03PS1) 1001tonythomas: AbuseFilter requires the AntiSpoof extension [integration/config] - 10https://gerrit.wikimedia.org/r/185634 (https://phabricator.wikimedia.org/T84859) [16:46:20] PROBLEM - Puppet failure on deployment-pdf01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [17:24:47] 3Phabricator: Actions shouldn't be attributed to bzimport - https://phabricator.wikimedia.org/T847#984012 (10Aklapper) 5Open>3stalled Until that happens, setting to Stalled status. [18:02:47] 3Phabricator: T87062 contains a centralauth token and other interesting cookie information allowing to steal session - https://phabricator.wikimedia.org/T87100#983917 (10Rillke) [18:11:23] I guess we should reset their token [18:11:24] sigh [18:40:18] Yippee, build fixed! [18:40:18] Project browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_8.1-internet_explorer-11-sauce build #258: FIXED in 33 min: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_8.1-internet_explorer-11-sauce/258/ [18:40:29] wikibugs is taking a while to announce things right now... [18:41:26] * greg-g goes [18:50:07] greg-g: tools-redis is out of diskspace, see -labs [18:54:44] Yippee, build fixed! [18:54:45] Project browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #484: FIXED in 43 min: https://integration.wikimedia.org/ci/job/browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/484/ [19:04:00] Yippee, build fixed! [19:04:01] Project browsertests-Echo-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #312: FIXED in 9 min 15 sec: https://integration.wikimedia.org/ci/job/browsertests-Echo-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/312/ [19:25:54] Yippee, build fixed! [19:25:55] Project browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-chrome-monobook-sauce build #242: FIXED in 34 min: https://integration.wikimedia.org/ci/job/browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-chrome-monobook-sauce/242/ [19:34:10] Yippee, build fixed! [19:34:10] Project browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce build #450: FIXED in 34 min: https://integration.wikimedia.org/ci/job/browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce/450/ [19:36:54] Yippee, build fixed! [19:36:55] Project browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-firefox-monobook-sauce build #258: FIXED in 43 min: https://integration.wikimedia.org/ci/job/browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-firefox-monobook-sauce/258/ [20:02:44] legoktm: Do you know how to fix the jenkins-stuck-queue-not-doing-executions thing? https://integration.wikimedia.org/ci/label/deployment-bastion-eqiad/load-statistics?type=hour [20:05:03] I...do not. [20:11:43] 3Beta-Cluster, Release-Engineering, MediaWiki-extensions-Flow, Collaboration-Team: beta-update-databases-eqiad failing on enwiki - https://phabricator.wikimedia.org/T86934#984119 (10Jdforrester-WMF) It appears that `deployment-bastion-eqiad` isn't executing anything (and hasn't since approximately 18:00 on 16 Ja... [20:14:07] legoktm: Oh well. [20:14:14] legoktm: Thanks. :-) [20:18:55] !log Marking deployment-bastion.eqiad as temporarily offline to try to fix the backlog. [20:19:02] Logged the message, Master [20:20:31] !log Brought deployment-bastion.eqiad back online, but without effect AFAICS. [20:20:34] Logged the message, Master [20:55:46] Project browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_8-internet_explorer-sauce build #411: FAILURE in 32 min: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_8-internet_explorer-sauce/411/ [21:57:21] (03PS2) 10Hashar: tests: factor out code to find defined project [integration/config] - 10https://gerrit.wikimedia.org/r/173830 [21:57:38] (03PS2) 10Hashar: (WIP) test Zuul gate configuration (WIP) [integration/config] - 10https://gerrit.wikimedia.org/r/173831 [21:58:59] (03CR) 10jenkins-bot: [V: 04-1] (WIP) test Zuul gate configuration (WIP) [integration/config] - 10https://gerrit.wikimedia.org/r/173831 (owner: 10Hashar)