[00:09:31] What is test-prio on zuul? [00:20:06] PROBLEM - Puppet run on deployment-phab02 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [00:22:55] PROBLEM - Puppet run on deployment-phab01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [00:24:01] thcipriani: umm something going on with phab ^ [00:26:52] Zppix: phab works fine for me? [00:27:20] I meant with puppet on phab01-02 [00:28:30] Zppix: what's the puppet error? [00:28:36] probably tell paladox [00:28:44] i assume he was testing stuff [00:28:46] Read shinken-wm_ [00:29:03] i don't see that, does it display the error to you? [00:30:07] 7:20 PM PROBLEM - Puppet run on deployment-phab02 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [00:30:22] aha, deployment-phab, not phab [00:30:26] that's a little different [00:30:34] i think [00:31:27] is that new-ish? [00:31:54] About 10 mins [00:32:04] i meant that this instance exists [00:32:08] ok [00:32:10] Oh idk [00:32:26] Thats just when shinken-wm_ reported an error [00:34:45] CRITICAL for Puppet run since 4d 12m [00:35:00] the web UI tells me these are broken since 4 days [00:35:14] so not that new [00:43:17] i cant SSH to those instances so could not lookup the actual puppet error [00:43:54] not sure why i can't. normally i can as root with my labs key and i am also in the project [00:44:10] anyways, since this isn't new or urgent and Friday almost 6pm, i'm moving on [01:25:48] (03PS1) 10Krinkle: mediawiki-core-code-coverage: Commit live hack [integration/config] - 10https://gerrit.wikimedia.org/r/344733 [01:26:01] (03CR) 10Krinkle: [C: 032] mediawiki-core-code-coverage: Commit live hack [integration/config] - 10https://gerrit.wikimedia.org/r/344733 (owner: 10Krinkle) [01:27:38] (03Merged) 10jenkins-bot: mediawiki-core-code-coverage: Commit live hack [integration/config] - 10https://gerrit.wikimedia.org/r/344733 (owner: 10Krinkle) [01:32:51] 10Continuous-Integration-Config, 10VisualEditor: VisualEditor-MediaWiki: Don't be "smart" about only running jsduck on JS file changes - https://phabricator.wikimedia.org/T155862#3129839 (10Jdforrester-WMF) p:05Triage>03Normal [03:03:54] 06Release-Engineering-Team, 06Operations, 05DC-Switchover-Prep-Q3-2016-17: Understand the preparedness of misc services for datacenter switchover - https://phabricator.wikimedia.org/T156937#3130017 (10Krinkle) [04:06:28] Project selenium-MultimediaViewer » firefox,beta,Linux,BrowserTests build #341: 04FAILURE in 10 min: https://integration.wikimedia.org/ci/job/selenium-MultimediaViewer/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=BrowserTests/341/ [06:51:11] Project selenium-Wikibase » chrome,beta,Linux,BrowserTests build #310: 04FAILURE in 2 hr 11 min: https://integration.wikimedia.org/ci/job/selenium-Wikibase/BROWSER=chrome,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=BrowserTests/310/ [10:02:12] PROBLEM - Host deployment-ores-redis-02 is DOWN: CRITICAL - Host Unreachable (10.68.18.121) [10:02:33] !log deleted deployment-ores-redis-02 [10:02:37] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [10:14:37] PROBLEM - Puppet run on deployment-ores-redis-01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [10:17:46] 10Continuous-Integration-Infrastructure, 06Labs, 10Labs-Infrastructure, 07Beta-Cluster-reproducible, 07Puppet: New instance have broken puppet configuration when using puppetmaster standalone - https://phabricator.wikimedia.org/T148929#2736876 (10Ladsgroup) I tried to work around this bug by doing the ex... [10:29:19] 10Continuous-Integration-Infrastructure, 06Labs, 10Labs-Infrastructure, 07Beta-Cluster-reproducible, 07Puppet: New instance have broken puppet configuration when using puppetmaster standalone - https://phabricator.wikimedia.org/T148929#3130632 (10Ladsgroup) After fighting with this for hours and thanks t... [10:31:51] 10Continuous-Integration-Infrastructure, 06Labs, 10Labs-Infrastructure, 07Beta-Cluster-reproducible, 07Puppet: New instance have broken puppet configuration when using puppetmaster standalone - https://phabricator.wikimedia.org/T148929#3130633 (10Ladsgroup) I might be wrong and I haven't tested this hypo... [10:39:38] RECOVERY - Puppet run on deployment-ores-redis-01 is OK: OK: Less than 1.00% above the threshold [0.0] [10:39:55] !log changing ores redis address to deployment-ores-redis-01 (T160762) [10:39:58] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [10:39:59] T160762: deployment-ores-redis /srv/ redis is too small (500MBytes) - https://phabricator.wikimedia.org/T160762 [10:46:17] !log deleting deployment-ores-redis (T160762) [10:46:20] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [10:46:20] T160762: deployment-ores-redis /srv/ redis is too small (500MBytes) - https://phabricator.wikimedia.org/T160762 [10:46:47] 10Beta-Cluster-Infrastructure, 10ORES, 06Revision-Scoring-As-A-Service, 15User-Ladsgroup: deployment-ores-redis /srv/ redis is too small (500MBytes) - https://phabricator.wikimedia.org/T160762#3130646 (10Ladsgroup) Okay. I migrated the redis server from deployment-ores-redis to deployment-ores-redis-01 whi... [10:47:25] 10Beta-Cluster-Infrastructure, 10ORES, 06Revision-Scoring-As-A-Service, 15User-Ladsgroup: deployment-ores-redis /srv/ redis is too small (500MBytes) - https://phabricator.wikimedia.org/T160762#3110103 (10Ladsgroup) 05Open>03Resolved [10:47:27] 10Beta-Cluster-Infrastructure, 10ORES, 06Revision-Scoring-As-A-Service, 13Patch-For-Review, 15User-Ladsgroup: Resurrect ores-beta with production roles - https://phabricator.wikimedia.org/T138445#3130651 (10Ladsgroup) [10:48:12] 10Continuous-Integration-Infrastructure, 06Labs, 10Labs-Infrastructure, 07Beta-Cluster-reproducible, 07Puppet: New instance have broken puppet configuration when using puppetmaster standalone - https://phabricator.wikimedia.org/T148929#3130652 (10Tarrow) I also suffered with this for a very long time. Ev... [10:48:54] PROBLEM - Host deployment-ores-redis is DOWN: CRITICAL - Host Unreachable (10.68.21.235) [10:49:15] I deleted it, why it's so stupid? [12:59:08] mutante Zppix hi, the test-prio pipeline on zuul for is for test to be higher priority then the other test pipeline [12:59:27] also hashar created the pipeline [13:05:27] (03CR) 10Paladox: "> We still have Wikimedia wikis running PHP 5.5." [integration/config] - 10https://gerrit.wikimedia.org/r/344642 (https://phabricator.wikimedia.org/T94149) (owner: 10Hashar) [14:33:41] Project selenium-WikiLove » firefox,beta,Linux,BrowserTests build #342: 04FAILURE in 1 min 40 sec: https://integration.wikimedia.org/ci/job/selenium-WikiLove/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=BrowserTests/342/ [15:43:49] (03PS1) 10Dereckson: Add Timeless skin [tools/release] - 10https://gerrit.wikimedia.org/r/344789 (https://phabricator.wikimedia.org/T160643) [15:44:39] Project selenium-MobileFrontend » chrome,beta,Linux,BrowserTests build #370: 04FAILURE in 22 min: https://integration.wikimedia.org/ci/job/selenium-MobileFrontend/BROWSER=chrome,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=BrowserTests/370/ [15:54:13] (03CR) 10Paladox: [C: 031] Add Timeless skin [tools/release] - 10https://gerrit.wikimedia.org/r/344789 (https://phabricator.wikimedia.org/T160643) (owner: 10Dereckson) [16:04:20] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 13Patch-For-Review: Depool precise jenkins instances - https://phabricator.wikimedia.org/T158652#3130891 (10Zppix) @hashar is still WIP or are we all done here? [17:10:40] 10Gerrit: Content of Openzim repository accidentially somehow deleted - https://phabricator.wikimedia.org/T161264#3130987 (10Paladox) @Kelson could you give it another go please? [17:13:14] 10Gerrit: Content of Openzim repository accidentially somehow deleted - https://phabricator.wikimedia.org/T161264#3130989 (10Kelson) @Paladox I have still exactly the same error (after merging your code). As far as I know this has always been a problem to import code from other people (without following the stan... [17:14:21] 10Gerrit: Content of Openzim repository accidentially somehow deleted - https://phabricator.wikimedia.org/T161264#3130990 (10Paladox) @Kelson ah, it was missing this one https://gerrit.wikimedia.org/r/#/c/344794/ could you merge it and retry please? [17:16:27] 10Gerrit: Content of Openzim repository accidentially somehow deleted - https://phabricator.wikimedia.org/T161264#3130991 (10Paladox) @Kelson did it work? [17:18:57] 10Gerrit: Content of Openzim repository accidentially somehow deleted - https://phabricator.wikimedia.org/T161264#3130993 (10Kelson) 05Open>03Resolved a:03Kelson @Paladox I have been able to reupload the github copy. For me it looks like the problem is fixed :) Thank you very much to the people involved in... [17:20:25] 10Gerrit: Content of Openzim repository accidentially somehow deleted - https://phabricator.wikimedia.org/T161264#3130996 (10Paladox) Your welcome :) [18:10:46] (03CR) 10Isarra: [C: 031] ":D" [tools/release] - 10https://gerrit.wikimedia.org/r/344789 (https://phabricator.wikimedia.org/T160643) (owner: 10Dereckson) [20:29:07] (03CR) 10Legoktm: "This is not necessary to deploy to beta cluster FWIW" [tools/release] - 10https://gerrit.wikimedia.org/r/344789 (https://phabricator.wikimedia.org/T160643) (owner: 10Dereckson) [20:38:06] (03Abandoned) 10Dereckson: Add Timeless skin [tools/release] - 10https://gerrit.wikimedia.org/r/344789 (https://phabricator.wikimedia.org/T160643) (owner: 10Dereckson)