[00:26:18] thanks, just a thought :) [00:26:30] legoktm lol :) [01:42:59] PROBLEM - Puppet staleness on deployment-logstash2 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [43200.0] [01:52:46] 10Beta-Cluster-Infrastructure, 10Sentry, 10User-Tgr: Integrate Sentry with beta cluster - https://phabricator.wikimedia.org/T106920#3807683 (10Tgr) Probably the Sentry config (specifically the API key) got reset during some maintenance of Cloud VPS boxes. I don't have the time to fix it right now; it should... [02:07:15] (03Draft2) 10Reedy: Remove `composer dump-autoload --optimize` [integration/jenkins] - 10https://gerrit.wikimedia.org/r/394907 [02:07:26] (03PS3) 10Reedy: Remove `composer dump-autoload --optimize` [integration/jenkins] - 10https://gerrit.wikimedia.org/r/394907 (https://phabricator.wikimedia.org/T181940) [02:09:02] 10Continuous-Integration-Infrastructure, 10Patch-For-Review: Address fixme in mw-fetch-composer-dev.sh - https://phabricator.wikimedia.org/T181940#3807690 (10Reedy) ``` $ php ../composer.phar dumpautoload --optimize Generating optimized autoload files ``` Seems to be a noop on an up to date repo... [03:36:04] PROBLEM - Free space - all mounts on deployment-fluorine02 is CRITICAL: CRITICAL: deployment-prep.deployment-fluorine02.diskspace._srv.byte_percentfree (<11.11%) [04:16:22] Yippee, build fixed! [04:16:23] Project selenium-MultimediaViewer » firefox,beta,Linux,BrowserTests build #597: 09FIXED in 20 min: https://integration.wikimedia.org/ci/job/selenium-MultimediaViewer/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=BrowserTests/597/ [04:52:50] PROBLEM - Mediawiki Error Rate on graphite-labs is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [10.0] [05:07:49] PROBLEM - Mediawiki Error Rate on graphite-labs is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [10.0] [06:45:33] 10Gerrit: Gerrit word diff is really bad at matching - https://phabricator.wikimedia.org/T181961#3807855 (10Nikerabbit) [07:11:02] RECOVERY - Free space - all mounts on deployment-fluorine02 is OK: OK: All targets OK [08:28:04] (03PS1) 10Hashar: Revert "Update composer to 1.4.3" [integration/composer] - 10https://gerrit.wikimedia.org/r/394930 [08:29:25] (03CR) 10Hashar: [C: 032] Revert "Update composer to 1.4.3" [integration/composer] - 10https://gerrit.wikimedia.org/r/394930 (owner: 10Hashar) [08:29:36] (03Merged) 10jenkins-bot: Revert "Update composer to 1.4.3" [integration/composer] - 10https://gerrit.wikimedia.org/r/394930 (owner: 10Hashar) [08:38:15] (03PS3) 10Robert Vogel: Changed settings for BlueSpice-repos [integration/config] - 10https://gerrit.wikimedia.org/r/394578 [08:42:14] (03CR) 10Robert Vogel: Changed settings for BlueSpice-repos (032 comments) [integration/config] - 10https://gerrit.wikimedia.org/r/394578 (owner: 10Robert Vogel) [08:45:04] (03PS4) 10Robert Vogel: Changed settings for BlueSpice-repos [integration/config] - 10https://gerrit.wikimedia.org/r/394578 [08:46:19] (03CR) 10jerkins-bot: [V: 04-1] Changed settings for BlueSpice-repos [integration/config] - 10https://gerrit.wikimedia.org/r/394578 (owner: 10Robert Vogel) [08:46:25] PROBLEM - Free space - all mounts on integration-slave-docker-1002 is CRITICAL: CRITICAL: integration.integration-slave-docker-1002.diskspace.root.byte_percentfree (<33.33%) [09:33:09] 10RelEng-Archive-FY201718-Q1, 10Release Pipeline (Blubber): Ensure application files are not copied for final multi-stage images - https://phabricator.wikimedia.org/T174623#3808005 (10Aklapper) [09:33:12] 10RelEng-Archive-FY201718-Q1, 10Release Pipeline (Blubber): Complete Blubber's support for multi-stage Dockerfiles - https://phabricator.wikimedia.org/T174620#3808006 (10Aklapper) [09:47:26] 10Release-Engineering-Team (Kanban), 10MediaWiki-Cache, 10MediaWiki-Vagrant, 10Performance-Team (Radar), 10User-zeljkofilipin: MediaWiki core Selenium tests fail when targeting Vagrant - https://phabricator.wikimedia.org/T180035#3808058 (10zeljkofilipin) @greg the only thing I did was report the problem.... [10:37:38] (03PS5) 10Robert Vogel: Changed settings for BlueSpice-repos [integration/config] - 10https://gerrit.wikimedia.org/r/394578 [11:23:01] RECOVERY - Puppet staleness on deployment-logstash2 is OK: OK: Less than 1.00% above the threshold [3600.0] [11:27:46] 10Continuous-Integration-Infrastructure, 10translatewiki.net, 10Patch-For-Review: L10n-bot should not force-merge / override Jenkins (breaks the build) - https://phabricator.wikimedia.org/T91707#1094080 (10Nikerabbit) > Yes, it sucks that our tests depend on the values of localisation messages in random lang... [12:18:06] PROBLEM - Free space - all mounts on integration-slave-jessie-1002 is CRITICAL: CRITICAL: integration.integration-slave-jessie-1002.diskspace._mnt.byte_percentfree (No valid datapoints found)integration.integration-slave-jessie-1002.diskspace._srv.byte_percentfree (<55.56%) [13:05:50] 10Release-Engineering-Team, 10ORES, 10Operations, 10Scoring-platform-team (Current): Connection timeout from tin to new ores servers - https://phabricator.wikimedia.org/T181661#3808791 (10akosiaris) >>! In T181661#3804344, @awight wrote: > I just ran scap with `-l "ores1001.*" and deployment went smoothly.... [14:45:47] 10Release-Engineering-Team, 10User-greg: Create #wikimedia-releng-feed and move bots there - https://phabricator.wikimedia.org/T181582#3809151 (10zeljkofilipin) @greg I ignore all bots in `#wikimedia-releng`. As far as I am concerned, we can remove IRC ping from Selenium jobs, or move them to another channel. [14:46:54] 10Release-Engineering-Team (Kanban), 10StructuredDiscussions, 10Browser-Tests, 10Collaboration-Team-Triage (Collab-Team-This-Quarter), and 2 others: Flow: Migrate browser tests from Ruby to node.js - https://phabricator.wikimedia.org/T174591#3809155 (10zeljkofilipin) a:05zeljkofilipin>03None [14:53:42] 10Release-Engineering-Team (Kanban), 10releng-201718-q1, 10MediaWiki-General-or-Unknown, 10Epic, and 5 others: Port Selenium tests from Ruby to Node.js - https://phabricator.wikimedia.org/T139740#3809169 (10zeljkofilipin) [15:15:02] PROBLEM - Puppet errors on deployment-redis06 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [15:26:28] PROBLEM - Free space - all mounts on integration-slave-docker-1002 is CRITICAL: CRITICAL: integration.integration-slave-docker-1002.diskspace.root.byte_percentfree (<11.11%) [15:27:44] PROBLEM - Puppet errors on deployment-redis01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [15:33:41] PROBLEM - Puppet errors on deployment-redis02 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [15:58:09] (03CR) 10Hashar: [C: 032] docker: add Chromedriver to npm-browser-test [integration/config] - 10https://gerrit.wikimedia.org/r/394565 (https://phabricator.wikimedia.org/T167507) (owner: 10Hashar) [16:00:57] (03Merged) 10jenkins-bot: docker: add Chromedriver to npm-browser-test [integration/config] - 10https://gerrit.wikimedia.org/r/394565 (https://phabricator.wikimedia.org/T167507) (owner: 10Hashar) [16:01:28] RECOVERY - Free space - all mounts on integration-slave-docker-1002 is OK: OK: All targets OK [16:23:07] (03PS1) 10Hashar: Typo for portals/deploy project name in Zuul [integration/config] - 10https://gerrit.wikimedia.org/r/395031 [16:31:58] 10Release-Engineering-Team (Watching / External), 10MediaWiki-Core-Tests, 10Patch-For-Review, 10User-zeljkofilipin: WebdriverIO should run Chrome headlessly - https://phabricator.wikimedia.org/T167507#3809537 (10hashar) 05stalled>03Open We now have a Docker image providing Chromium 62. [16:34:50] (03CR) 10Hashar: [C: 032] "Sorry that it was set a bit too short. I have updated the job." [integration/config] - 10https://gerrit.wikimedia.org/r/394824 (https://phabricator.wikimedia.org/T181881) (owner: 10Dalba) [16:36:16] (03Merged) 10jenkins-bot: Increase pywikibot-core-tox-doc-docker timeout to 5 minutes [integration/config] - 10https://gerrit.wikimedia.org/r/394824 (https://phabricator.wikimedia.org/T181881) (owner: 10Dalba) [16:49:00] 10Beta-Cluster-Infrastructure, 10Scoring-platform-team, 10Wikimedia-Logstash, 10monitoring: Make an ORES service log dashboard for logstash-beta - https://phabricator.wikimedia.org/T182005#3809625 (10awight) [16:54:53] 10Beta-Cluster-Infrastructure, 10Scoring-platform-team, 10Wikimedia-Logstash, 10monitoring: Make an ORES service log dashboard for logstash-beta - https://phabricator.wikimedia.org/T182005#3809669 (10awight) Currently, apache loglines are tagged with type "ores", e.g. > [pid: 10481] 10.68.21.68 (-) {34 var... [17:02:09] (03CR) 10Hashar: "Maybe. I am tempted to instead have it enabled globally so we make sure we never miss adding --jobs." [integration/config] - 10https://gerrit.wikimedia.org/r/391712 (owner: 10Chad) [17:08:13] 10Release-Engineering-Team, 10ORES, 10Operations, 10Scoring-platform-team (Current): Connection timeout from tin to new ores servers - https://phabricator.wikimedia.org/T181661#3809754 (10mmodell) >>! In T181661#3808791, @akosiaris wrote: > > So, scap closed the connection, 1m, 6s after the login. scap c... [17:17:30] (03CR) 10Xqt: "Seems 5' aren't enough:" [integration/config] - 10https://gerrit.wikimedia.org/r/394824 (https://phabricator.wikimedia.org/T181881) (owner: 10Dalba) [17:23:01] 10Release-Engineering-Team (Kanban), 10User-zeljkofilipin: Replace jshint with eslint in nodemw - https://phabricator.wikimedia.org/T181285#3809911 (10zeljkofilipin) 05Open>03Resolved Merged upstream. [17:37:18] (03CR) 10Chad: "Doing it via /etc/gitconfig is also an option, I honestly didn't even think of that route" [integration/config] - 10https://gerrit.wikimedia.org/r/391712 (owner: 10Chad) [17:46:22] 10Release-Engineering-Team (Watching / External), 10Scap, 10ORES, 10Operations, 10Scoring-platform-team: scap support for git-lfs - https://phabricator.wikimedia.org/T181855#3809968 (10demon) > What will happen if we try to checkout a project with git-lfs-enabled submodules on tin? Scap does not speak g... [17:47:50] 10Release-Engineering-Team (Kanban), 10Scap (Tech Debt Sprint FY201718-Q2), 10Scoring-platform-team: Need to make the number of cached revisions configurable - https://phabricator.wikimedia.org/T181176#3809969 (10mmodell) 05Open>03Resolved [17:48:22] PROBLEM - Puppet errors on deployment-mediawiki05 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [17:50:23] PROBLEM - Puppet errors on integration-slave-jessie-1002 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [17:51:35] 10Release-Engineering-Team (Watching / External), 10Scap, 10ORES, 10Operations, 10Scoring-platform-team: scap support for git-lfs - https://phabricator.wikimedia.org/T181855#3809993 (10awight) >> And will scap be able to fetch and checkout on deployment targets? > > See above. The only caveat is that I'... [17:52:53] PROBLEM - Puppet errors on deployment-jobrunner02 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [18:01:07] 10Release-Engineering-Team (Watching / External), 10Scap, 10ORES, 10Operations, 10Scoring-platform-team: scap support for git-lfs - https://phabricator.wikimedia.org/T181855#3810031 (10demon) Phab's already behind Varnish, Gerrit is not yet (cf some bug I don't have in front of me). But with objects this... [18:04:08] 10Release-Engineering-Team (Kanban), 10Scap (Tech Debt Sprint FY201718-Q2), 10scap2, 10Patch-For-Review: Eliminate symlinks in mediawiki-config (as much as possible) - https://phabricator.wikimedia.org/T126306#3810039 (10mmodell) Is this one resolved now? I know you made some progress this quarter. [18:04:44] 10Release-Engineering-Team (Watching / External), 10Scap, 10ORES, 10Operations, 10Scoring-platform-team: scap support for git-lfs - https://phabricator.wikimedia.org/T181855#3810042 (10awight) Good point, that won't work at all. On the bright side, we know that the git-lfs load is actually smaller than... [18:10:49] RECOVERY - Free space - all mounts on deployment-sca03 is OK: OK: All targets OK [18:13:32] 10Release-Engineering-Team (Watching / External), 10Scap, 10ORES, 10Operations, 10Scoring-platform-team: scap support for git-lfs - https://phabricator.wikimedia.org/T181855#3810070 (10demon) Yep I think we're on the same page here. [18:16:40] Just wanted to bring this task up i created over the weekend incase it wasnt noticed... T181899 [18:16:41] T181899: mwgate-php55lint fails to fetch origin - https://phabricator.wikimedia.org/T181899 [18:21:07] 10Release-Engineering-Team (Kanban), 10Scap (Tech Debt Sprint FY201718-Q2), 10scap2, 10Patch-For-Review: Eliminate symlinks in mediawiki-config (as much as possible) - https://phabricator.wikimedia.org/T126306#3810086 (10demon) p:05High>03Low Not quite, it's still a WIP there's a long tail. [18:21:56] greg-g: FYI this is ready for review, I thought I would ping you well in advance of our meeting on Thursday… https://wikitech.wikimedia.org/wiki/ORES/Deployment [18:22:41] Mostly, I added some plaster and gravel to the monitoring and unhappy paths. [18:28:21] RECOVERY - Puppet errors on deployment-mediawiki05 is OK: OK: Less than 1.00% above the threshold [0.0] [18:31:31] (03PS1) 10Chad: You basically should never use special_extensions. [tools/release] - 10https://gerrit.wikimedia.org/r/395060 [18:32:52] RECOVERY - Puppet errors on deployment-jobrunner02 is OK: OK: Less than 1.00% above the threshold [0.0] [18:39:23] PROBLEM - Puppet errors on deployment-mx is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [19:06:00] Project beta-scap-eqiad build #184849: 04FAILURE in 2 min 8 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/184849/ [19:08:11] Project beta-scap-eqiad build #184850: 04STILL FAILING in 2 min 8 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/184850/ [19:09:41] 19:05:59 UnicodeDecodeError: 'utf8' codec can't decode byte 0xd0 in position 176: invalid continuation byte [19:09:41] 19:05:59 19:05:59 cdb-json-refresh failed: 'utf8' codec can't decode byte 0xd0 in position 176: invalid continuation byte [19:09:42] scap bug? [19:11:48] Project selenium-MinervaNeue » firefox,beta,Linux,BrowserTests build #224: 15ABORTED in 22 min: https://integration.wikimedia.org/ci/job/selenium-MinervaNeue/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=BrowserTests/224/ [19:12:09] Project beta-scap-eqiad build #184851: 04STILL FAILING in 2 min 9 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/184851/ [19:12:44] twentyafterfour: ^^ [19:15:40] Project beta-scap-eqiad build #184852: 04STILL FAILING in 1 min 55 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/184852/ [19:17:42] Project beta-scap-eqiad build #184853: 04STILL FAILING in 1 min 58 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/184853/ [19:25:35] Project beta-scap-eqiad build #184854: 04STILL FAILING in 1 min 56 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/184854/ [19:28:49] RECOVERY - Mediawiki Error Rate on graphite-labs is OK: OK: Less than 1.00% above the threshold [1.0] [19:31:33] Project beta-scap-eqiad build #184855: 04STILL FAILING in 1 min 54 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/184855/ [19:35:35] Project beta-scap-eqiad build #184856: 04STILL FAILING in 1 min 55 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/184856/ [19:39:30] 10Release-Engineering-Team (Watching / External), 10Epic, 10MediaWiki-Platform-Team (MWPT-Q2-Oct-Dec-2017): Deploy refactored comment storage - https://phabricator.wikimedia.org/T166733#3810425 (10Anomie) [19:42:46] Project selenium-MinervaNeue » firefox,beta,Linux,BrowserTests build #225: 04STILL FAILING in 30 min: https://integration.wikimedia.org/ci/job/selenium-MinervaNeue/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=BrowserTests/225/ [19:45:40] Project beta-scap-eqiad build #184857: 04STILL FAILING in 1 min 57 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/184857/ [19:46:15] Hey, I commited a patch to Gerrit which works with Firefox and Chrome on my local vagrant machine, but Selenium is failing the test, even when the change isn't related to my patch. Is selenium preferring src above srcset? [19:46:17] https://integration.wikimedia.org/ci/job/mwskin-mw-selenium-jessie/830/console [19:49:51] PROBLEM - Mediawiki Error Rate on graphite-labs is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [10.0] [19:55:43] Project beta-scap-eqiad build #184858: 04STILL FAILING in 2 min 0 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/184858/ [20:05:48] Project beta-scap-eqiad build #184859: 04STILL FAILING in 2 min 7 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/184859/ [20:07:58] Project beta-scap-eqiad build #184860: 04STILL FAILING in 2 min 6 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/184860/ [20:08:57] 10Continuous-Integration-Config, 10Pywikibot-core, 10Patch-For-Review, 10Pywikibot-tests: pywikibot-core-tox-doc-docker timeouts sporadically - https://phabricator.wikimedia.org/T181881#3810583 (10hashar) 05Open>03Resolved a:03Dalba @Dalba managed to find the root cause and bumped the job timeout fro... [20:08:59] 20:07:57 20:07:57 cdb-json-refresh failed: 'utf8' codec can't decode byte 0xd0 in position 176: invalid continuation byte [20:09:04] Just to confirm: We’re welcome to run schema changes on the beta cluster using update.php? [20:09:24] Ummmm, jenkins/scap tasks do that for you [20:09:44] I'm pretty sure I have $wgDontRunUpdateDotPHPImSeriousLeaveItAlone set on beta too [20:09:50] (or whatever we called the variable) [20:10:40] (03CR) 10Krinkle: [C: 031] Swap node for jq [integration/jenkins] - 10https://gerrit.wikimedia.org/r/394856 (https://phabricator.wikimedia.org/T181938) (owner: 10Reedy) [20:12:01] (03CR) 10Krinkle: [C: 031] ""deprecated" also works :)" [tools/release] - 10https://gerrit.wikimedia.org/r/395060 (owner: 10Chad) [20:15:57] Project beta-scap-eqiad build #184861: 04STILL FAILING in 2 min 14 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/184861/ [20:23:59] Hmmm, trying to run that locally on deployment-tin gives me nothing nicer in terms of output. [20:24:04] I wonder if some weird message was merged.... [20:25:44] Project beta-scap-eqiad build #184862: 04STILL FAILING in 1 min 56 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/184862/ [20:25:46] no_justification: you mean we don't have it set, right? Doesn't beta use update.php by default for its updates? [20:25:57] Yeah, it does. But humans shouldn't need to [20:25:59] I thought it ran from the jenkins job every time. [20:26:01] Right right [20:26:15] I thought the variable we have affects the script in general, not humans only. [20:26:22] So we wouldn't haave it set I guess. [20:26:26] unless there is an override :P [20:26:43] MW_NO_REALLY_REALLY_REALLY_DO_DO_RUN_UPDATE_PHP [20:33:13] legoktm: weird... [20:35:36] Project beta-scap-eqiad build #184863: 04STILL FAILING in 1 min 57 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/184863/ [20:42:59] Project selenium-Echo » chrome,beta,Linux,BrowserTests build #599: 04FAILURE in 1 min 59 sec: https://integration.wikimedia.org/ci/job/selenium-Echo/BROWSER=chrome,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=BrowserTests/599/ [20:45:05] (03CR) 10Reedy: [C: 032] "/me waits for you to hunt fundraising down" [tools/release] - 10https://gerrit.wikimedia.org/r/395060 (owner: 10Chad) [20:45:35] Project beta-scap-eqiad build #184864: 04STILL FAILING in 1 min 53 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/184864/ [20:45:44] (03Merged) 10jenkins-bot: You basically should never use special_extensions. [tools/release] - 10https://gerrit.wikimedia.org/r/395060 (owner: 10Chad) [20:46:09] 10Continuous-Integration-Infrastructure, 10translatewiki.net: L10n-bot should not force-merge / override Jenkins (breaks the build) - https://phabricator.wikimedia.org/T91707#3810707 (10Krinkle) [20:47:01] PROBLEM - Free space - all mounts on deployment-fluorine02 is CRITICAL: CRITICAL: deployment-prep.deployment-fluorine02.diskspace._srv.byte_percentfree (<20.00%) [20:48:24] (03CR) 10Awight: "LOL, see bug T179536." [tools/release] - 10https://gerrit.wikimedia.org/r/395060 (owner: 10Chad) [20:54:49] RECOVERY - Mediawiki Error Rate on graphite-labs is OK: OK: Less than 1.00% above the threshold [1.0] [20:55:37] Project beta-scap-eqiad build #184865: 04STILL FAILING in 1 min 57 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/184865/ [20:59:30] thcipriani: I'm kinda at a loss ^ [20:59:42] the jenkins output matches what I get on disk [20:59:45] It's barfing on some utf8 [21:00:01] hrm [21:00:12] wonder if some weird message or comment merged... [21:00:34] error is not that helpful since we're doing some weird thread things here IIRC [21:04:03] I'm not seeing any suspicious messages, but hmm. [21:05:39] Project beta-scap-eqiad build #184866: 04STILL FAILING in 1 min 59 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/184866/ [21:05:42] looks like it's failing on: /srv/mediawiki-staging/php-master/cache/l10n/l10n_cache-tyv.cdb [21:06:10] and it's failing in json.dumps... [21:11:49] 10Continuous-Integration-Infrastructure, 10translatewiki.net: L10n-bot should not force-merge / override Jenkins (breaks the build) - https://phabricator.wikimedia.org/T91707#3810787 (10matmarex) It probably wouldn't be hard, but it would be tedious, since you'd probably have to update the expected values in e... [21:13:04] (03PS1) 10Hashar: docker: set FORCE_COLOR for supports-color [integration/config] - 10https://gerrit.wikimedia.org/r/395093 [21:15:40] Project beta-scap-eqiad build #184867: 04STILL FAILING in 2 min 0 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/184867/ [21:15:48] (03CR) 10Hashar: "Came around it while looking at the webdriver.io tests being added to wikimedia/portals/deploy https://gerrit.wikimedia.org/r/#/c/394539/" [integration/config] - 10https://gerrit.wikimedia.org/r/395093 (owner: 10Hashar) [21:18:57] (03CR) 10Hashar: "We dont have jq installed. On a Nodepool instance:" [integration/jenkins] - 10https://gerrit.wikimedia.org/r/394856 (https://phabricator.wikimedia.org/T181938) (owner: 10Reedy) [21:23:25] no_justification: this is a utf-8 problem, I think. If you do '\xd0\xb0'.decode('utf-8') it's fine, but if you do something like '\xd0'.decode('utf-8') it explodes because the first byte denotes a 2-byte utf-8 codepoint. If the 2nd byte doesn't match 10xxxxxx it'll explode. Looking at https://en.wikipedia.org/wiki/UTF-8#Design [21:24:03] Did someone write a malformed message? [21:24:21] so there's some problem in /srv/mediawiki-staging/php-master/cache/l10n/l10n_cache-tyv.cdb where \xd0 isn't followed by a 10xxxxxx message (I *think*) [21:25:19] (03CR) 10Hashar: "Sounds like a good idea. Lets first:" [integration/jenkins] - 10https://gerrit.wikimedia.org/r/394856 (https://phabricator.wikimedia.org/T181938) (owner: 10Reedy) [21:25:43] no_justification: I think it's in namespacenames: \xd0\xa2 [21:25:43] Project beta-scap-eqiad build #184868: 04STILL FAILING in 2 min 2 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/184868/ [21:26:10] I can't wait for more jobs to be running on the docker nodes to speed up this ci [21:27:13] (03CR) 10Addshore: [C: 031] docker: set FORCE_COLOR for supports-color [integration/config] - 10https://gerrit.wikimedia.org/r/395093 (owner: 10Hashar) [21:27:52] (03CR) 10Addshore: [C: 031] Migrate some npm jobs to Docker [integration/config] - 10https://gerrit.wikimedia.org/r/393246 (owner: 10Hashar) [21:28:04] (03PS1) 10Hashar: Swap a nodejs oneliner to jq [integration/config] - 10https://gerrit.wikimedia.org/r/395100 (https://phabricator.wikimedia.org/T181938) [21:28:45] (03CR) 10Hashar: "That is the same as https://gerrit.wikimedia.org/r/#/c/394856/ :D" [integration/config] - 10https://gerrit.wikimedia.org/r/395100 (https://phabricator.wikimedia.org/T181938) (owner: 10Hashar) [21:30:44] (03PS5) 10Hashar: Swap node for jq [integration/jenkins] - 10https://gerrit.wikimedia.org/r/394856 (https://phabricator.wikimedia.org/T181938) (owner: 10Reedy) [21:34:59] 10Release-Engineering-Team, 10ORES, 10Operations, 10Scoring-platform-team (Current): Connection timeout from tin to new ores servers - https://phabricator.wikimedia.org/T181661#3810860 (10mmodell) [21:40:30] woo i got log4j2 working with gerrit https://gerrit-review.googlesource.com/#/c/gerrit/+/142811/ :). I can now specify custom params we can use in logstash to identify gerrit. (only works with log4j 2+). [21:40:41] 10Continuous-Integration-Infrastructure, 10Continuous-Integration-Scaling, 10releng-201415-Q3, 10releng-201415-Q4, and 2 others: [EPIC] Run CI jobs in disposable VMs - https://phabricator.wikimedia.org/T47499#3810867 (10hashar) [21:40:44] 10Continuous-Integration-Scaling, 10Release-Engineering-Team (Someday): Isolate contintcloud nova project from the rest of the wmflabs cloud - https://phabricator.wikimedia.org/T86168#3810865 (10hashar) 05Open>03declined That is no more needed. #nodepool is legacy and it will be phased out (as well as `con... [21:43:09] (03PS1) 10Hashar: Remove wikidata/gremlink job [integration/config] - 10https://gerrit.wikimedia.org/r/395113 (https://phabricator.wikimedia.org/T155829) [21:44:34] (03CR) 10Smalyshev: [C: 031] Remove wikidata/gremlink job [integration/config] - 10https://gerrit.wikimedia.org/r/395113 (https://phabricator.wikimedia.org/T155829) (owner: 10Hashar) [21:45:01] (03PS2) 10Smalyshev: Remove wikidata/gremlin job [integration/config] - 10https://gerrit.wikimedia.org/r/395113 (https://phabricator.wikimedia.org/T155829) (owner: 10Hashar) [21:48:48] Yippee, build fixed! [21:48:48] Project beta-scap-eqiad build #184869: 09FIXED in 15 min: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/184869/ [21:51:03] Hey CI experts, is it possible to have jenkins autopull merged changes ?? [21:51:14] On toolforge [21:57:12] Zppix: you can create jobs to do just about anything [21:58:44] PROBLEM - Puppet errors on saucelabs-01 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [22:02:00] PROBLEM - puppet last run on contint2001 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [22:04:17] PROBLEM - Puppet errors on saucelabs-03 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [22:08:09] PROBLEM - puppet last run on contint1001 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [22:09:39] PROBLEM - Puppet errors on saucelabs-02 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [22:13:09] RECOVERY - puppet last run on contint1001 is OK: OK: Puppet is currently enabled, last run 22 seconds ago with 0 failures [22:27:20] (03CR) 10Krinkle: [C: 04-1] "This is basically for consumption by Jenkins, right? Can Jenkins set these at run-time? I thought it did that already. but I guess those a" [integration/config] - 10https://gerrit.wikimedia.org/r/395093 (owner: 10Hashar) [22:30:23] (03CR) 10Krinkle: [C: 04-1] "@Hashar: Almost. I think mw-* is used by mw-related jobs, and the unprefixed one for jobs like for mediawiki-config. I created the variant" [integration/config] - 10https://gerrit.wikimedia.org/r/395100 (https://phabricator.wikimedia.org/T181938) (owner: 10Hashar) [22:32:00] RECOVERY - puppet last run on contint2001 is OK: OK: Puppet is currently enabled, last run 4 minutes ago with 0 failures [22:33:43] RECOVERY - Puppet errors on saucelabs-01 is OK: OK: Less than 1.00% above the threshold [0.0] [22:39:08] could someone check gerrit logs if there is more info related to this error?: [22:39:11] error: fatal: BatchRefUpdate failed: BatchRefUpdate[ UPDATE: e424a64d4c792cfd807d276ef7f94a9a1bf78499 75e96100f6944e4e971f27b48a6fadbed791a5db refs/heads/master (LOCK_FAILURE) [22:39:16] ] [22:39:18] RECOVERY - Puppet errors on saucelabs-03 is OK: OK: Less than 1.00% above the threshold [0.0] [22:44:39] RECOVERY - Puppet errors on saucelabs-02 is OK: OK: Less than 1.00% above the threshold [0.0] [23:20:19] Nikerabbit hi, you can view the logs in logstash if you have access now :) [23:28:41] 10Release-Engineering-Team (Watching / External), 10Epic, 10MediaWiki-Platform-Team (MWPT-Q2-Oct-Dec-2017): Deploy refactored comment storage - https://phabricator.wikimedia.org/T166733#3305978 (10Krinkle) >>! In T166733#3810388, @gerritbot wrote: > Change 395073 merged by jenkins-bot: > [operations/mediawik... [23:29:26] greg-g, the deployment calendar can remove g.wicke's nick probably [23:29:31] s/probably// [23:30:57] PROBLEM - Puppet errors on deployment-kafka-jumbo-2 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [23:31:21] Nikerabbit i think the problem is the same as https://phabricator.wikimedia.org/T155558 [23:31:23] 10Release-Engineering-Team (Watching / External), 10Operations, 10User-Joe: [DRAFT][RfC] Deployment of python applications in production - https://phabricator.wikimedia.org/T180023#3811081 (10greg) [23:32:03] subbu: heh, right :) [23:36:10] PROBLEM - Puppet errors on deployment-netbox is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [23:37:26] PROBLEM - Puppet errors on deployment-kafka-jumbo-1 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0]