[00:05:35] oh wow it does it for all jobs [00:05:52] i can jjwrecker -s https://integration.wikimedia.org/ci/ [00:15:37] (03PS1) 10Paladox: Create operations-puppet-catalog-compiler job in jjb [integration/config] - 10https://gerrit.wikimedia.org/r/325064 (https://phabricator.wikimedia.org/T97513) [00:15:50] (03PS2) 10Paladox: Create operations-puppet-catalog-compiler job in jjb [integration/config] - 10https://gerrit.wikimedia.org/r/325064 (https://phabricator.wikimedia.org/T97513) [00:16:46] (03PS3) 10Paladox: Create operations-puppet-catalog-compiler job in jjb [integration/config] - 10https://gerrit.wikimedia.org/r/325064 (https://phabricator.wikimedia.org/T97513) [00:17:43] 10Continuous-Integration-Config, 07Technical-Debt: Migrate "analytics-*" jobs to Jenkins Job Builder - https://phabricator.wikimedia.org/T97514#2843212 (10Paladox) I've found a new tool https://github.com/ktdreyer/jenkins-job-wrecker that will make the seemless, needs a jenkins admin to run the request though. [00:33:22] 03Scap3 (Scap3-MediaWiki-MVP), 03releng-201617-q2: Flatten MediaWiki config, all MediaWiki versions, and extensions into a unified git repo - https://phabricator.wikimedia.org/T147478#2693801 (10greg) \o/ https://www.mediawiki.org/w/index.php?title=Wikimedia_Release_Engineering_Team%2FGoals%2F201617Q2&type=re... [01:40:31] greg-g: FYI I'm going to do an emergency deploy in the next 15ish minutes to fix article lead images being missing for most enwiki articles in the mobile app [01:41:07] (https://gerrit.wikimedia.org/r/#/c/325075/) [02:21:10] twentyafterfour: rsync between iridium and phab2001 works now. and we avoided hardcoding any IPs or names, just based on the hiera setting which is the active server [02:21:34] i deleted the old contents of /srv/repos on phab2001, and now a fresh sync is running on iridium in screen [02:22:48] *waves*, nice weekend [02:34:07] mutante: awesome! [03:25:44] 06Release-Engineering-Team, 06Operations, 10Phabricator, 13Patch-For-Review: Setup test domain for phab2001 - https://phabricator.wikimedia.org/T152132#2843493 (10Dzahn) p:05Triage>03Normal [04:12:15] Project selenium-MultimediaViewer » firefox,beta,Linux,contintLabsSlave && UbuntuTrusty build #222: 04FAILURE in 16 min: https://integration.wikimedia.org/ci/job/selenium-MultimediaViewer/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=contintLabsSlave%20&&%20UbuntuTrusty/222/ [06:41:46] PROBLEM - Puppet run on deployment-restbase01 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [06:51:43] Yippee, build fixed! [06:51:43] Project selenium-Wikibase » chrome,beta,Linux,contintLabsSlave && UbuntuTrusty build #196: 09FIXED in 2 hr 11 min: https://integration.wikimedia.org/ci/job/selenium-Wikibase/BROWSER=chrome,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=contintLabsSlave%20&&%20UbuntuTrusty/196/ [07:16:45] RECOVERY - Puppet run on deployment-restbase01 is OK: OK: Less than 1.00% above the threshold [0.0] [10:44:37] (03PS1) 10Legoktm: Whitelist Filip / rlot [integration/config] - 10https://gerrit.wikimedia.org/r/325093 [10:44:38] (03PS1) 10Legoktm: [ContentTranslation] Remove jshint [integration/config] - 10https://gerrit.wikimedia.org/r/325094 [10:46:21] (03CR) 10Legoktm: [C: 032] Whitelist Filip / rlot [integration/config] - 10https://gerrit.wikimedia.org/r/325093 (owner: 10Legoktm) [10:46:34] (03CR) 10Legoktm: [C: 032] [ContentTranslation] Remove jshint [integration/config] - 10https://gerrit.wikimedia.org/r/325094 (owner: 10Legoktm) [10:47:12] (03Merged) 10jenkins-bot: Whitelist Filip / rlot [integration/config] - 10https://gerrit.wikimedia.org/r/325093 (owner: 10Legoktm) [10:47:30] (03Merged) 10jenkins-bot: [ContentTranslation] Remove jshint [integration/config] - 10https://gerrit.wikimedia.org/r/325094 (owner: 10Legoktm) [10:48:32] !log deploying https://gerrit.wikimedia.org/r/325093 and https://gerrit.wikimedia.org/r/325094 [10:48:36] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [15:02:12] (03Abandoned) 10Paladox: Enable zuul/jenkins to run unit tests in skin projects [integration/config] - 10https://gerrit.wikimedia.org/r/228470 (https://phabricator.wikimedia.org/T68926) (owner: 10Paladox) [15:39:49] PROBLEM - Puppet run on deployment-phab02 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [15:51:05] PROBLEM - Puppet run on deployment-phab01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [17:08:01] (03PS1) 10Paladox: Add a puppet compiler pipeline [integration/config] - 10https://gerrit.wikimedia.org/r/325109 [17:09:04] (03PS4) 10Paladox: Create operations-puppet-catalog-compiler job in jjb [integration/config] - 10https://gerrit.wikimedia.org/r/325064 (https://phabricator.wikimedia.org/T97513) [17:09:13] (03PS2) 10Paladox: Add a puppet compiler pipeline [integration/config] - 10https://gerrit.wikimedia.org/r/325109 [17:12:14] (03PS5) 10Paladox: Create operations-puppet-catalog-compiler job in jjb [integration/config] - 10https://gerrit.wikimedia.org/r/325064 (https://phabricator.wikimedia.org/T97513) [17:12:24] (03PS3) 10Paladox: Add a puppet compiler pipeline [integration/config] - 10https://gerrit.wikimedia.org/r/325109 [18:51:51] (03PS6) 10Paladox: Create operations-puppet-catalog-compiler job in jjb [integration/config] - 10https://gerrit.wikimedia.org/r/325064 (https://phabricator.wikimedia.org/T97513) [18:52:27] (03PS4) 10Paladox: Add a puppet compiler pipeline [integration/config] - 10https://gerrit.wikimedia.org/r/325109 [18:53:12] (03PS5) 10Paladox: Add a puppet compiler pipeline [integration/config] - 10https://gerrit.wikimedia.org/r/325109 [18:53:21] (03CR) 10jenkins-bot: [V: 04-1] Create operations-puppet-catalog-compiler job in jjb [integration/config] - 10https://gerrit.wikimedia.org/r/325064 (https://phabricator.wikimedia.org/T97513) (owner: 10Paladox) [18:53:55] (03PS7) 10Paladox: Create operations-puppet-catalog-compiler job in jjb [integration/config] - 10https://gerrit.wikimedia.org/r/325064 (https://phabricator.wikimedia.org/T97513) [18:54:27] (03PS6) 10Paladox: Add a puppet compiler pipeline [integration/config] - 10https://gerrit.wikimedia.org/r/325109 [18:55:26] (03CR) 10jenkins-bot: [V: 04-1] Create operations-puppet-catalog-compiler job in jjb [integration/config] - 10https://gerrit.wikimedia.org/r/325064 (https://phabricator.wikimedia.org/T97513) (owner: 10Paladox) [19:02:31] (03PS8) 10Paladox: Create operations-puppet-catalog-compiler job in jjb [integration/config] - 10https://gerrit.wikimedia.org/r/325064 (https://phabricator.wikimedia.org/T97513) [19:02:53] (03CR) 10Paladox: "recheck" [integration/config] - 10https://gerrit.wikimedia.org/r/325064 (https://phabricator.wikimedia.org/T97513) (owner: 10Paladox) [19:02:58] (03PS7) 10Paladox: Add a puppet compiler pipeline [integration/config] - 10https://gerrit.wikimedia.org/r/325109 [19:19:27] (03PS9) 10Paladox: Create operations-puppet-catalog-compiler job in jjb [integration/config] - 10https://gerrit.wikimedia.org/r/325064 (https://phabricator.wikimedia.org/T97513) [19:19:33] (03PS8) 10Paladox: Add a puppet compiler pipeline [integration/config] - 10https://gerrit.wikimedia.org/r/325109 [19:20:48] (03PS14) 10Paladox: Support extension and skin dependacies in the skin pipeline and extension pipeline [integration/config] - 10https://gerrit.wikimedia.org/r/323540 (https://phabricator.wikimedia.org/T151593) [19:28:44] (03PS15) 10Paladox: Support extension and skin dependacies in the skin pipeline and extension pipeline [integration/config] - 10https://gerrit.wikimedia.org/r/323540 (https://phabricator.wikimedia.org/T151593) [19:29:26] (03CR) 10Paladox: "@Hashar oh so this works but is hacky?" [integration/config] - 10https://gerrit.wikimedia.org/r/323540 (https://phabricator.wikimedia.org/T151593) (owner: 10Paladox) [21:25:28] PROBLEM - Long lived cherry-picks on puppetmaster on deployment-puppetmaster02 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [22:49:14] legoktm im wondering should we whitelist all GCI16 users on wikimedia temporarily? [22:49:41] Most of the tests were switched to nodepool so there is low risk of anyone compromsing wikimedia servers. [22:50:06] paladox: if they have 2 or more merged patches, let's whitelist them [22:50:10] Ok [22:50:17] thanks [22:53:23] 06Release-Engineering-Team, 06Operations, 06Parsing-Team, 07HHVM, and 3 others: API cluster failure / OOM - https://phabricator.wikimedia.org/T151702#2828601 (10Theklan) Hello, I'm Theklan, one of the admins of eu:wp. We have been making some chages in Modulu:Wikidata, in order to fully use Wikidata's pow... [22:54:19] (03PS1) 10Paladox: Whitelist GCI16 student Divadsn temporarily [integration/config] - 10https://gerrit.wikimedia.org/r/325132 [22:54:23] legoktm ^^ :) [22:54:28] he has two or more changes merged [22:54:33] https://gerrit.wikimedia.org/r/#/q/owner:%22Divadsn+%253Cdivad.nnamtdeis%2540gmail.com%253E%22 [22:55:04] paladox: ok, also not temporarily, I think we can just leave them whitelisted, there's already a section for GCI students [22:55:12] Ok [22:55:15] Oh [22:55:31] ah found it [22:55:32] thanks [22:56:18] (03PS2) 10Paladox: Whitelist GCI16 student Divadsn [integration/config] - 10https://gerrit.wikimedia.org/r/325132 [22:56:22] legoktm ^^ done [22:56:23] :) [22:59:17] paladox: ¿you added him twice? [22:59:24] Yeh it is needed [22:59:56] Platonides it's to block him from being tested in the check pipeline, since we want the test pipeline not the check pipeline [23:00:20] line 85 and line 474 :P [23:00:25] All users who are whitelisted are blocked from that pipeline [23:00:36] looks like only one would be enough [23:00:37] Yeh, 474 is the test pipeline [23:00:42] and line 85? [23:00:45] 85 is the check pipeline [23:01:07] oh, right [23:01:10] Yep [23:01:12] they're different sections [23:01:15] yep [23:01:41] there are two duplicate email lists :( [23:02:07] :( [23:02:09] Yep, soon to be removed [23:02:20] I'm seeing it on the comments: [23:02:21] # TODO: Figure out a way to not have to repeat this from pipeline 'test'. [23:02:24] # This email filter and the one for 'test' can be removed once we have fixed bug 45499. [23:02:26] that's is if we decide to stick with nodepool or go with nodepool [23:02:34] in fact, it should probably fetch most info from ldap [23:03:05] it seems to be able to create a kind of template [23:03:06] &email_whitelist [23:03:12] that you can copy to the other pipelines [23:03:45] oh, indeed [23:03:51] the file is yaml [23:03:57] But the syntax is different for the check pipeline [23:04:00] and yaml supports that syntax for referencing other parts [23:04:05] as it is blocking the emails with those addreses [23:04:10] yep [23:04:44] Platonides: well one list is a string (pipe separated regex) and the other is an actual list [23:04:55] Upstream zuul are dropping support for jenkins, doint know why they carn't use there new system and keep support for jenkins. [23:05:12] (03CR) 10Legoktm: [C: 032] Whitelist GCI16 student Divadsn [integration/config] - 10https://gerrit.wikimedia.org/r/325132 (owner: 10Paladox) [23:05:21] Thanks legoktm ^^ [23:05:40] good point, legoktm [23:05:50] why, oh why? [23:06:00] (03Merged) 10jenkins-bot: Whitelist GCI16 student Divadsn [integration/config] - 10https://gerrit.wikimedia.org/r/325132 (owner: 10Paladox) [23:07:46] !log deploying https://gerrit.wikimedia.org/r/325132 [23:07:49] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [23:15:40] Platonides: Ideally we just want to get rid of the whitelist entirely and let everyone run the full suite of tests [23:15:50] that's already the case for pywikibot [23:18:38] what's blocking it? [23:19:11] Platonides the nodepool migration [23:19:33] labs does not have enough resources for nodepool, we only last week got bumped to 20 nodepool instances [23:19:49] Note that they are created and deleted on the fly. They are not static instances. [23:20:05] change the 20 into 19 due to a bug in nodepool [23:20:11] we had to downgrade by 1. [23:20:37] Platonides hashar is slowly upgrading all the tests to nodepool anyways :) [23:28:56] hmmm, ok [23:37:52] legoktm im wondering would putting the test pipeline emails into a | work? [23:39:04] email: &email_whitelist [23:39:04] (.*?@wikimedia\.org| .*?@wikimedia\.de) [23:39:08] something like ^^ [23:41:00] you seem to have an extra space in front of the .de case [23:41:13] but it would be simpler to do .*?@wikimedia\.(org|de) [23:41:19] Oh thansk [23:41:21] thanks [23:43:20] (03PS1) 10Paladox: Updating regex for the email so that we do org|de and not org and de seperatly [integration/config] - 10https://gerrit.wikimedia.org/r/325137 [23:43:28] Krenair legoktm ^^ :) [23:51:08] (03PS1) 10Paladox: Update email regex part 2 [integration/config] - 10https://gerrit.wikimedia.org/r/325140 [23:51:22] Platonides legoktm ^^ i am not sure if that will work [23:52:41] (03CR) 10jenkins-bot: [V: 04-1] Update email regex part 2 [integration/config] - 10https://gerrit.wikimedia.org/r/325140 (owner: 10Paladox) [23:53:11] (03PS2) 10Paladox: Update email regex part 2 [integration/config] - 10https://gerrit.wikimedia.org/r/325140 [23:54:59] (03PS3) 10Paladox: Update email regex part 2 [integration/config] - 10https://gerrit.wikimedia.org/r/325140 [23:56:35] (03CR) 10jenkins-bot: [V: 04-1] Update email regex part 2 [integration/config] - 10https://gerrit.wikimedia.org/r/325140 (owner: 10Paladox) [23:57:46] jenkins disliked it