[00:01:42] PROBLEM - Free space - all mounts on deployment-fluorine02 is CRITICAL: CRITICAL: deployment-prep.deployment-fluorine02.diskspace._srv.byte_percentfree (<55.56%) [00:11:29] RECOVERY - Mediawiki Error Rate on graphite-labs is OK: OK: Less than 1.00% above the threshold [1.0] [00:14:41] (03PS7) 10Niedzielski: Update chromium-render to use Debian Chromium [integration/config] - 10https://gerrit.wikimedia.org/r/409115 (https://phabricator.wikimedia.org/T179552) [00:15:32] (03CR) 10Niedzielski: "Thanks @hashar! Revised!" (032 comments) [integration/config] - 10https://gerrit.wikimedia.org/r/409115 (https://phabricator.wikimedia.org/T179552) (owner: 10Niedzielski) [00:31:16] PROBLEM - Long lived cherry-picks on puppetmaster on deployment-puppetmaster02 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [00:51:10] PROBLEM - English Wikipedia Main page on beta-cluster is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 24722 bytes in 3.265 second response time [00:51:24] PROBLEM - App Server Main HTTP Response on deployment-mediawiki04 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 24169 bytes in 3.229 second response time [00:53:08] PROBLEM - App Server Main HTTP Response on deployment-mediawiki05 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 24164 bytes in 3.209 second response time [00:54:46] PROBLEM - English Wikipedia Mobile Main page on beta-cluster is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 19571 bytes in 3.237 second response time [00:55:16] PROBLEM - App Server Main HTTP Response on deployment-mediawiki06 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 24189 bytes in 3.992 second response time [00:59:29] Hm.. default dashboard on logstash-beta is gone? [01:04:48] RECOVERY - English Wikipedia Mobile Main page on beta-cluster is OK: HTTP OK: HTTP/1.1 200 OK - 36216 bytes in 3.689 second response time [01:05:20] RECOVERY - App Server Main HTTP Response on deployment-mediawiki06 is OK: HTTP OK: HTTP/1.1 200 OK - 47134 bytes in 5.727 second response time [01:06:14] RECOVERY - English Wikipedia Main page on beta-cluster is OK: HTTP OK: HTTP/1.1 200 OK - 47719 bytes in 4.152 second response time [01:06:26] RECOVERY - App Server Main HTTP Response on deployment-mediawiki04 is OK: HTTP OK: HTTP/1.1 200 OK - 47174 bytes in 3.832 second response time [01:08:09] RECOVERY - App Server Main HTTP Response on deployment-mediawiki05 is OK: HTTP OK: HTTP/1.1 200 OK - 47124 bytes in 5.630 second response time [01:13:24] ^ was me [01:16:35] 10Beta-Cluster-Infrastructure: [betalabs] Fatal exception of type "Wikimedia\Rdbms\DBQueryError" on any page - https://phabricator.wikimedia.org/T187416#3974468 (10Etonkovidova) [01:35:22] 10Beta-Cluster-Infrastructure: [betalabs] Fatal exception of type "Wikimedia\Rdbms\DBQueryError" on any page - https://phabricator.wikimedia.org/T187416#3974505 (10MaxSem) 05Open>03Resolved a:03MaxSem My bad, was fixed before this was posted. [01:38:46] (03CR) 10Krinkle: [C: 032] OOjs: Drop -jsduck-docker job, repo moving to jsdoc [integration/config] - 10https://gerrit.wikimedia.org/r/410255 (owner: 10Jforrester) [01:38:56] (03PS3) 10Krinkle: OOjs: Drop -jsduck-docker job, repo moving to jsdoc [integration/config] - 10https://gerrit.wikimedia.org/r/410255 (owner: 10Jforrester) [01:41:34] !log Reloading Zuul to deploy https://gerrit.wikimedia.org/r/410255 [01:41:39] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [01:54:12] 10Phabricator (2018-02-xx), 10DBA, 10Release: Upcoming phabricator upgrade requires unusually long database migrations - https://phabricator.wikimedia.org/T187143#3974522 (10mmodell) [01:54:16] 10Phabricator, 10Release-Engineering-Team (Kanban), 10Operations, 10Patch-For-Review, 10User-Elukey: Apache on phab1001 is gradually leaking worker processes which are stuck in "Gracefully finishing" state - https://phabricator.wikimedia.org/T182832#3974521 (10mmodell) [01:56:11] 10Phabricator (2018-02-15), 10DBA, 10Release: Upcoming phabricator upgrade requires unusually long database migrations - https://phabricator.wikimedia.org/T187143#3974524 (10mmodell) [01:56:14] 10Phabricator (2018-02-15), 10Upstream: allow users to export maniphest advanced search to csv - https://phabricator.wikimedia.org/T103009#3974526 (10mmodell) [01:56:16] 10Phabricator (2018-02-15), 10Upstream: When another user removes you as a subscriber from a task, you don't receive an email notification - https://phabricator.wikimedia.org/T126711#3974525 (10mmodell) [02:10:29] 10Diffusion, 10Gerrit, 10Phabricator: Delete all Phabricator git repos that haven't been referenced / aren't used. - https://phabricator.wikimedia.org/T187149#3974534 (10demon) Gerrit also provides multi-datacenter capability, so we've got local mirrors. I understand maybe wanting a non-Gerrit mirror we cont... [02:25:24] 10Phabricator, 10Patch-For-Review, 10Performance: /maniphest/report/project/ : Maximum execution time of 10 seconds exceeded - https://phabricator.wikimedia.org/T125357#3974539 (10Paladox) I've raised the timeout here ^^ hoping that will at least improve it some what, but this does seem very slow. [02:28:53] 10Phabricator (2018-02-15), 10Upstream: Distinguish "mentions" from "subscribers" - https://phabricator.wikimedia.org/T150766#3974540 (10mmodell) [02:39:31] PROBLEM - Puppet errors on deployment-memc06 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [02:41:15] 10Diffusion, 10Gerrit, 10Phabricator: Delete all Phabricator git repos that haven't been referenced / aren't used. - https://phabricator.wikimedia.org/T187149#3974545 (10mmodell) >>! In T187149#3974534, @demon wrote: > Now... I'd be fine with nuking them and recreating them in a sensible manner, dropping the... [02:45:04] 10Phabricator, 10Release-Engineering-Team (Kanban), 10Operations, 10Patch-For-Review, 10User-Elukey: Apache on phab1001 is gradually leaking worker processes which are stuck in "Gracefully finishing" state - https://phabricator.wikimedia.org/T182832#3974546 (10mmodell) This looks much better now: ``` _... [02:46:57] 10Phabricator, 10Release-Engineering-Team (Kanban), 10Operations, 10Patch-For-Review, 10User-Elukey: Apache on phab1001 is gradually leaking worker processes which are stuck in "Gracefully finishing" state - https://phabricator.wikimedia.org/T182832#3974547 (10Dzahn) I fixed the server status page. it's... [03:01:13] 10Phabricator (2018-02-15), 10DBA, 10Release: Upcoming phabricator upgrade requires unusually long database migrations - https://phabricator.wikimedia.org/T187143#3974548 (10mmodell) [03:06:32] 10releng-201617-q3, 10Gerrit-Migration, 10Goal: Phase 2 repository migrations to Differential - https://phabricator.wikimedia.org/T130420#3974551 (10demon) 05stalled>03declined Declining outright as we don't have any sort of plan here. There's no "phase 2" and the suggested repos are non-viable (I know o... [03:19:29] RECOVERY - Puppet errors on deployment-memc06 is OK: OK: Less than 1.00% above the threshold [0.0] [04:49:31] PROBLEM - Mediawiki Error Rate on graphite-labs is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [10.0] [05:14:30] PROBLEM - Mediawiki Error Rate on graphite-labs is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [10.0] [05:51:46] 10Phabricator (2018-02-15), 10DBA, 10Release: Upcoming phabricator upgrade requires unusually long database migrations - https://phabricator.wikimedia.org/T187143#3974649 (10jcrespo) That is not what I asked, I asked you to disable the first one (the alter), not the second one. [06:02:35] (03PS3) 10Legoktm: Release 16.0.0 [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/410535 (owner: 10Jforrester) [06:07:36] (03PS4) 10Legoktm: Release 16.0.0 [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/410535 (owner: 10Jforrester) [06:15:49] PROBLEM - Free space - all mounts on deployment-eventlog02 is CRITICAL: CRITICAL: deployment-prep.deployment-eventlog02.diskspace.root.byte_percentfree (<33.33%) [06:18:27] (03CR) 10Legoktm: [C: 032] "I regenerated the changelog with the utils/gen-changelog.sh script, which sorts the entries for consistency. Thanks :)" [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/410535 (owner: 10Jforrester) [06:19:32] (03Merged) 10jenkins-bot: Release 16.0.0 [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/410535 (owner: 10Jforrester) [06:20:56] (03CR) 10jenkins-bot: Release 16.0.0 [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/410535 (owner: 10Jforrester) [06:28:17] 10Phabricator (2018-02-15), 10DBA, 10Patch-For-Review, 10Release: Upcoming phabricator upgrade requires unusually long database migrations - https://phabricator.wikimedia.org/T187143#3974673 (10mmodell) [06:57:03] (03PS1) 10Legoktm: Only run extension patch coverage if a *.php file was changed [integration/config] - 10https://gerrit.wikimedia.org/r/410654 [07:00:49] RECOVERY - Free space - all mounts on deployment-eventlog02 is OK: OK: All targets OK [07:10:43] (03CR) 10Legoktm: [C: 032] Only run extension patch coverage if a *.php file was changed [integration/config] - 10https://gerrit.wikimedia.org/r/410654 (owner: 10Legoktm) [07:12:16] (03Merged) 10jenkins-bot: Only run extension patch coverage if a *.php file was changed [integration/config] - 10https://gerrit.wikimedia.org/r/410654 (owner: 10Legoktm) [07:12:33] PROBLEM - Puppet errors on saucelabs-01 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [07:13:03] !log deployed https://gerrit.wikimedia.org/r/410654 [07:13:08] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [07:16:42] RECOVERY - Free space - all mounts on deployment-fluorine02 is OK: OK: All targets OK [07:26:21] !log cancelled a job of mediawiki-core-doxygen-publish to unstuck them [07:26:38] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [07:30:51] 10Phabricator (2018-02-15), 10DBA, 10Patch-For-Review, 10Release: Upcoming phabricator upgrade requires unusually long database migrations - https://phabricator.wikimedia.org/T187143#3974703 (10mmodell) 05Open>03Resolved a:03jcrespo Everything looks good, thanks @jcrespo and @Marostegui [07:30:53] 10Phabricator, 10Release-Engineering-Team (Kanban), 10Operations, 10Patch-For-Review, 10User-Elukey: Apache on phab1001 is gradually leaking worker processes which are stuck in "Gracefully finishing" state - https://phabricator.wikimedia.org/T182832#3974706 (10mmodell) [07:30:59] 10Diffusion, 10Gerrit, 10Patch-For-Review: Commits merged in Gerrit should appear near-instantly in Phabricator - https://phabricator.wikimedia.org/T183792#3974709 (10Legoktm) I filed this task originally since it was causing problems for libraryupgrader, but that has now switched to gitiles (T187150) so thi... [07:31:02] 10Phabricator (2018-02-15), 10DBA, 10Patch-For-Review, 10Release: Upcoming phabricator upgrade requires unusually long database migrations - https://phabricator.wikimedia.org/T187143#3965756 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by jynus on neodymium.eqiad.wmnet for hosts: ``` ['dbpro... [07:34:39] 10Phabricator, 10Release-Engineering-Team (Kanban), 10Operations, 10Patch-For-Review, 10User-Elukey: Apache on phab1001 is gradually leaking worker processes which are stuck in "Gracefully finishing" state - https://phabricator.wikimedia.org/T182832#3974735 (10mmodell) Ok so the upstream code is deployed... [07:44:34] 10Phabricator (2018-02-15), 10DBA, 10Patch-For-Review, 10Release: Upcoming phabricator upgrade requires unusually long database migrations - https://phabricator.wikimedia.org/T187143#3974755 (10ops-monitoring-bot) Completed auto-reimage of hosts: ``` ['dbproxy1003.eqiad.wmnet'] ``` and were **ALL** succes... [07:47:10] (03PS1) 10Hashar: debian-glue: hook dir might not exist [integration/config] - 10https://gerrit.wikimedia.org/r/410659 (https://phabricator.wikimedia.org/T187405) [07:47:35] RECOVERY - Puppet errors on saucelabs-01 is OK: OK: Less than 1.00% above the threshold [0.0] [07:47:40] 10Diffusion, 10Gerrit, 10Patch-For-Review: Commits merged in Gerrit should appear near-instantly in Phabricator - https://phabricator.wikimedia.org/T183792#3974760 (10mmodell) @legoktm: So although I think we should still do some things to make phabricator update faster, given that the original need is now m... [07:49:23] 10Release-Engineering-Team (Kanban), 10Scap, 10Patch-For-Review: Scap: TypeError: error: (not all arguments converted during string formatting); format string: (Passed unrecognized git_binary_manager {}); arguments: ((u'fat',)) - https://phabricator.wikimedia.org/T184882#3974762 (10mmodell) can this be close... [07:49:56] (03CR) 10Hashar: [C: 032] debian-glue: hook dir might not exist [integration/config] - 10https://gerrit.wikimedia.org/r/410659 (https://phabricator.wikimedia.org/T187405) (owner: 10Hashar) [07:51:22] (03Merged) 10jenkins-bot: debian-glue: hook dir might not exist [integration/config] - 10https://gerrit.wikimedia.org/r/410659 (https://phabricator.wikimedia.org/T187405) (owner: 10Hashar) [07:52:04] 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure, 10Patch-For-Review: jenkins-debian-glue is failing for unstable - https://phabricator.wikimedia.org/T187405#3974765 (10hashar) 05Open>03Resolved a:03hashar The hook directories are generated by the puppet module `package_builder`... [08:51:17] 10Diffusion, 10Gerrit, 10Phabricator: Delete all Phabricator git repos that haven't been referenced / aren't used. - https://phabricator.wikimedia.org/T187149#3974820 (10Legoktm) [09:05:02] twentyafterfour: thank you for https://phabricator.wikimedia.org/phame/post/view/85/phabricator_updates_for_february_2018/ and there are definitely a few new features I will totally love (mail stamps, tasks close date) [09:13:32] 10Phabricator, 10Release-Engineering-Team (Kanban), 10Operations, 10Patch-For-Review, 10User-Elukey: Apache on phab1001 is gradually leaking worker processes which are stuck in "Gracefully finishing" state - https://phabricator.wikimedia.org/T182832#3974829 (10elukey) >>! In T182832#3974547, @Dzahn wrote... [09:40:56] 10Phabricator, 10Release-Engineering-Team (Kanban), 10Operations, 10Patch-For-Review, 10User-Elukey: Apache on phab1001 is gradually leaking worker processes which are stuck in "Gracefully finishing" state - https://phabricator.wikimedia.org/T182832#3974852 (10mmodell) @elukey: last night, before the upg... [09:50:24] (03PS1) 10Hashar: Migrate maven-site-publish jobs to Docker [integration/config] - 10https://gerrit.wikimedia.org/r/410768 [10:03:07] 10Phabricator, 10Release-Engineering-Team (Kanban), 10Operations, 10Patch-For-Review, 10User-Elukey: Apache on phab1001 is gradually leaking worker processes which are stuck in "Gracefully finishing" state - https://phabricator.wikimedia.org/T182832#3974890 (10mmodell) I'm now convinced that the problem... [10:06:01] 10Phabricator, 10Release-Engineering-Team (Kanban), 10Operations, 10Patch-For-Review, 10User-Elukey: Apache on phab1001 is gradually leaking worker processes which are stuck in "Gracefully finishing" state - https://phabricator.wikimedia.org/T182832#3974891 (10elukey) >>! In T182832#3974890, @mmodell wro... [10:17:23] PROBLEM - Free space - all mounts on integration-slave-jessie-1001 is CRITICAL: CRITICAL: integration.integration-slave-jessie-1001.diskspace._mnt.byte_percentfree (No valid datapoints found)integration.integration-slave-jessie-1001.diskspace._srv.byte_percentfree (<11.11%) [10:35:11] !log gerrit: marking search/ltr and search/repository-swift as read-only | T187428 [10:35:17] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [10:35:17] T187428: search/repository-swift and search/ltr fail on "mvn site": Unable to connect to: http://maven.elasticsearch.org/releases - https://phabricator.wikimedia.org/T187428 [10:36:39] RECOVERY - Free space - all mounts on integration-slave-jessie-1002 is OK: OK: integration.integration-slave-jessie-1002.diskspace._mnt.byte_percentfree (No valid datapoints found) [10:37:25] PROBLEM - Free space - all mounts on integration-slave-jessie-1001 is CRITICAL: CRITICAL: integration.integration-slave-jessie-1001.diskspace._mnt.byte_percentfree (No valid datapoints found)integration.integration-slave-jessie-1001.diskspace._srv.byte_percentfree (<44.44%) [10:37:26] (03PS1) 10Hashar: Archive search/ltr and search/swift-repository [integration/config] - 10https://gerrit.wikimedia.org/r/410793 (https://phabricator.wikimedia.org/T187428) [10:37:45] dcausse: merci pour ltr / swift-repository. Deux problΓ¨mes de moins :] [10:37:55] hashar: de rien! :) [10:38:08] (03CR) 10Hashar: [C: 032] Archive search/ltr and search/swift-repository [integration/config] - 10https://gerrit.wikimedia.org/r/410793 (https://phabricator.wikimedia.org/T187428) (owner: 10Hashar) [10:48:03] (03CR) 10jerkins-bot: [V: 04-1] Archive search/ltr and search/swift-repository [integration/config] - 10https://gerrit.wikimedia.org/r/410793 (https://phabricator.wikimedia.org/T187428) (owner: 10Hashar) [10:52:25] RECOVERY - Free space - all mounts on integration-slave-jessie-1001 is OK: OK: integration.integration-slave-jessie-1001.diskspace._mnt.byte_percentfree (No valid datapoints found) [10:54:07] PROBLEM - Host deployment-videoscaler01 is DOWN: CRITICAL - Host Unreachable (10.68.19.130) [10:54:52] PROBLEM - Host deployment-tmh01 is DOWN: CRITICAL - Host Unreachable (10.68.16.211) [11:06:13] 10Phabricator (2018-02-15), 10Upstream: Distinguish "mentions" from "subscribers" - https://phabricator.wikimedia.org/T150766#3975007 (10Elitre) >>! In T150766#3971531, @mmodell wrote: > Mail Stamps will soon be exposed in the body of messages, rather than only in the X-Phabricator-Mail-Stamps header. This sho... [11:19:39] 10Phabricator (2018-02-15), 10Upstream: Distinguish "mentions" from "subscribers" - https://phabricator.wikimedia.org/T150766#3975048 (10mmodell) 05Open>03Resolved a:03mmodell Mail stamps are available now. The feature is controlled by `Settings` β†’ `Email Format` β†’ `Send Stamps` The upstream documenta... [11:30:58] 10Continuous-Integration-Infrastructure, 10Patch-For-Review, 10Release Pipeline (Blubber): jenkins-slave@contint1001 not a member of docker group (CI tests for mathoid broken) - https://phabricator.wikimedia.org/T186790#3975108 (10akosiaris) 05Open>03Resolved a:03akosiaris I don't particularly love the... [11:55:19] 10Phabricator, 10Patch-For-Review, 10Performance: /maniphest/report/project/ : Maximum execution time of 10 seconds exceeded - https://phabricator.wikimedia.org/T125357#3975203 (10Aklapper) @Paladox: Please read Chad's comment from 2017-02-06 on https://gerrit.wikimedia.org/r/#/c/335714/ again. Ignoring fee... [12:12:55] 10Phabricator, 10Release-Engineering-Team (Kanban), 10Operations, 10Patch-For-Review, 10User-Elukey: Apache on phab1001 is gradually leaking worker processes which are stuck in "Gracefully finishing" state - https://phabricator.wikimedia.org/T182832#3975241 (10Paladox) @elukey what about backporting php7... [12:27:07] 10Continuous-Integration-Infrastructure, 10Patch-For-Review, 10Release Pipeline (Blubber): jenkins-slave@contint1001 not a member of docker group (CI tests for mathoid broken) - https://phabricator.wikimedia.org/T186790#3975260 (10Physikerwelt) @akosiaris Thank you very much for fixing that. When will the pa... [12:27:43] 10Gerrit: Bring diffusion links back - https://phabricator.wikimedia.org/T187439#3975261 (10MarcoAurelio) [12:30:53] PROBLEM - Puppet errors on deployment-ores01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [12:32:21] 10Phabricator, 10Patch-For-Review, 10Performance: /maniphest/report/project/ : Maximum execution time of 10 seconds exceeded - https://phabricator.wikimedia.org/T125357#3975273 (10Paladox) @Aklapper sorry, i forgot about that, i cannot remember everything. [12:35:04] 10Continuous-Integration-Infrastructure, 10Patch-For-Review, 10Release Pipeline (Blubber): jenkins-slave@contint1001 not a member of docker group (CI tests for mathoid broken) - https://phabricator.wikimedia.org/T186790#3975275 (10akosiaris) It already has been. jenkins-slave is part of that group currently.... [12:36:23] Project mwext-phpunit-coverage-publish build #975: 04FAILURE in 38 sec: https://integration.wikimedia.org/ci/job/mwext-phpunit-coverage-publish/975/ [13:24:24] (03CR) 10Hashar: [C: 032] Archive search/ltr and search/swift-repository [integration/config] - 10https://gerrit.wikimedia.org/r/410793 (https://phabricator.wikimedia.org/T187428) (owner: 10Hashar) [13:30:02] (03Merged) 10jenkins-bot: Archive search/ltr and search/swift-repository [integration/config] - 10https://gerrit.wikimedia.org/r/410793 (https://phabricator.wikimedia.org/T187428) (owner: 10Hashar) [13:35:56] Yippee, build fixed! [13:35:57] Project mwext-phpunit-coverage-publish build #976: 09FIXED in 35 sec: https://integration.wikimedia.org/ci/job/mwext-phpunit-coverage-publish/976/ [13:53:13] 10Continuous-Integration-Config, 10Security-Team, 10phan-taint-check-plugin, 10MediaWiki-Platform-Team (MWPT-Q3-Jan-Mar-2018), 10Patch-For-Review: Make jenkins run security-check-plugin non-voting - https://phabricator.wikimedia.org/T182599#3975418 (10Bawolff) Ok, fixes for the failures at: https://gerri... [14:05:52] (03PS2) 10Hashar: Migrate most maven-site-publish jobs to Docker [integration/config] - 10https://gerrit.wikimedia.org/r/410768 [14:06:49] (03PS1) 10Hashar: Remove unused JJB variable for search jobs [integration/config] - 10https://gerrit.wikimedia.org/r/410912 [14:11:16] (03CR) 10Hashar: [C: 032] Migrate most maven-site-publish jobs to Docker [integration/config] - 10https://gerrit.wikimedia.org/r/410768 (owner: 10Hashar) [14:11:40] (03CR) 10Hashar: [C: 032] "noop in JJB as expected" [integration/config] - 10https://gerrit.wikimedia.org/r/410912 (owner: 10Hashar) [14:13:48] (03Merged) 10jenkins-bot: Migrate most maven-site-publish jobs to Docker [integration/config] - 10https://gerrit.wikimedia.org/r/410768 (owner: 10Hashar) [14:14:47] (03Merged) 10jenkins-bot: Remove unused JJB variable for search jobs [integration/config] - 10https://gerrit.wikimedia.org/r/410912 (owner: 10Hashar) [14:24:19] (03Abandoned) 10Hashar: Convert all composer images to docker-pkg [integration/config] - 10https://gerrit.wikimedia.org/r/388452 (owner: 10Giuseppe Lavagetto) [14:24:33] (03Abandoned) 10Hashar: Convert mediawiki-* images to docker-pkg [integration/config] - 10https://gerrit.wikimedia.org/r/391564 (owner: 10Giuseppe Lavagetto) [14:53:16] PROBLEM - Host deployment-puppetdb01 is DOWN: CRITICAL - Host Unreachable (10.68.23.76) [15:21:03] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban), 10Patch-For-Review: Swap node for jq in mw-fetch-composer-dev.sh - https://phabricator.wikimedia.org/T181938#3975664 (10hashar) [15:21:05] 10Continuous-Integration-Infrastructure (shipyard), 10Release-Engineering-Team (Kanban): Migrate operations-mw-config-composer-hhvm-jessie to Docker - https://phabricator.wikimedia.org/T186145#3975663 (10hashar) [15:25:18] 10Continuous-Integration-Infrastructure (shipyard), 10Release-Engineering-Team (Kanban): Migrate operations-mw-config-composer-hhvm-jessie to Docker - https://phabricator.wikimedia.org/T186145#3975686 (10hashar) The dev dependencies are not included in `multiversion/vendor` though at least they are in `compose... [15:25:46] (03PS1) 10Hashar: Migrate mw-config composer job to Docker [integration/config] - 10https://gerrit.wikimedia.org/r/410950 (https://phabricator.wikimedia.org/T181938) [15:37:27] (03PS2) 10Hashar: Migrate mw-config composer job to Docker [integration/config] - 10https://gerrit.wikimedia.org/r/410950 (https://phabricator.wikimedia.org/T186145) [15:37:51] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban), 10Patch-For-Review: Swap node for jq in mw-fetch-composer-dev.sh - https://phabricator.wikimedia.org/T181938#3975754 (10hashar) >>! In T181938#3975688, @gerritbot wrote: > Change 410950 had a related patch set uploaded (by Hashar;... [16:02:30] 10Phabricator, 10Release-Engineering-Team (Kanban), 10Operations, 10Patch-For-Review, 10User-Elukey: Apache on phab1001 is gradually leaking worker processes which are stuck in "Gracefully finishing" state - https://phabricator.wikimedia.org/T182832#3975871 (10mmodell) Paladin: how many packages from bus... [16:02:53] (03PS1) 10Hashar: docker: add jq to composer images [integration/config] - 10https://gerrit.wikimedia.org/r/410990 (https://phabricator.wikimedia.org/T186145) [16:33:17] PROBLEM - Puppet errors on deployment-logstash2 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [16:45:14] (03PS2) 10Hashar: docker: script to only install composer dev dependencies [integration/config] - 10https://gerrit.wikimedia.org/r/410990 (https://phabricator.wikimedia.org/T186145) [16:45:47] PROBLEM - Free space - all mounts on deployment-mediawiki04 is CRITICAL: CRITICAL: deployment-prep.deployment-mediawiki04.diskspace.root.byte_percentfree (<11.11%) [16:49:06] (03PS3) 10Hashar: docker: script to only install composer dev dependencies [integration/config] - 10https://gerrit.wikimedia.org/r/410990 (https://phabricator.wikimedia.org/T186145) [16:50:05] (03PS4) 10Hashar: docker: script to only install composer dev dependencies [integration/config] - 10https://gerrit.wikimedia.org/r/410990 (https://phabricator.wikimedia.org/T186145) [16:52:05] 10Continuous-Integration-Infrastructure (shipyard), 10Release-Engineering-Team (Kanban), 10Patch-For-Review: Migrate operations-mw-config-composer-hhvm-jessie to Docker - https://phabricator.wikimedia.org/T186145#3976148 (10hashar) a:03hashar [16:53:37] 10Continuous-Integration-Infrastructure (shipyard), 10Release-Engineering-Team (Kanban), 10Patch-For-Review: Migrate operations-mw-config-composer-hhvm-jessie to Docker - https://phabricator.wikimedia.org/T186145#3935015 (10hashar) p:05Triage>03High [16:53:47] 10Gerrit, 10Release-Engineering-Team (Kanban), 10Patch-For-Review, 10Zuul: Update zuul to latest pre 3.0 commit - https://phabricator.wikimedia.org/T158243#3976168 (10hashar) p:05Low>03Lowest [17:06:56] 10Phabricator (2018-02-15), 10Upstream: Distinguish "mentions" from "subscribers" - https://phabricator.wikimedia.org/T150766#3976238 (10Aklapper) Added a sentence to https://www.mediawiki.org/w/index.php?title=Phabricator%2FHelp&type=revision&diff=2715419&oldid=2708150 Also see https://secure.phabricator.com... [17:19:43] 10Phabricator, 10Release-Engineering-Team (Kanban), 10Operations, 10Patch-For-Review, 10User-Elukey: Apache on phab1001 is gradually leaking worker processes which are stuck in "Gracefully finishing" state - https://phabricator.wikimedia.org/T182832#3976280 (10Paladox) @mmodell hi, i have a list here htt... [17:22:15] 10Phabricator, 10Release-Engineering-Team (Kanban), 10Operations, 10Patch-For-Review, 10User-Elukey: Apache on phab1001 is gradually leaking worker processes which are stuck in "Gracefully finishing" state - https://phabricator.wikimedia.org/T182832#3976287 (10mmodell) Looks like we still have workers in... [17:22:46] 10Phabricator, 10Release-Engineering-Team (Kanban), 10Operations, 10Patch-For-Review, 10User-Elukey: Apache on phab1001 is gradually leaking worker processes which are stuck in "Gracefully finishing" state - https://phabricator.wikimedia.org/T182832#3976290 (10Paladox) @mmodell full list https://phabrica... [17:25:44] PROBLEM - English Wikipedia Mobile Main page on beta-cluster is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Service Unavailable - string 'Wikipedia' not found on 'https://en.m.wikipedia.beta.wmflabs.org:443/wiki/Main_Page?debug=true' - 1997 bytes in 0.593 second response time [17:30:46] RECOVERY - English Wikipedia Mobile Main page on beta-cluster is OK: HTTP OK: HTTP/1.1 200 OK - 36213 bytes in 4.249 second response time [17:41:20] 10Diffusion, 10Gerrit, 10Phabricator: Delete all Phabricator git repos that haven't been referenced / aren't used. - https://phabricator.wikimedia.org/T187149#3965942 (10greg) Personally, I like having the repos in Phab; I don't dislike Diffusion as much as some others do and the clean markup for linking/ren... [17:53:48] 10Phabricator, 10Release-Engineering-Team (Kanban), 10Operations, 10Patch-For-Review, 10User-Elukey: Apache on phab1001 is gradually leaking worker processes which are stuck in "Gracefully finishing" state - https://phabricator.wikimedia.org/T182832#3976453 (10elukey) Got a backtrace of one process in G... [17:56:45] 10Phabricator, 10Release-Engineering-Team (Kanban), 10Operations, 10Patch-For-Review, 10User-Elukey: Apache on phab1001 is gradually leaking worker processes which are stuck in "Gracefully finishing" state - https://phabricator.wikimedia.org/T182832#3976455 (10Paladox) Should we change serialize_precisio... [18:39:17] SMalyshev for your 12 point on https://lists.wikimedia.org/pipermail/wikitech-l/2018-February/089517.html [18:39:29] the WIP status badge is in gerrit master (2.16 / 3.0) [18:39:37] coolio1 [18:39:39] WIP status is added in gerrit 2.15 [18:39:44] *! [18:39:45] in 2.14, wip is not supported. [18:40:03] so eventually we'll get it [18:40:06] great [18:40:09] yep [18:40:18] SMalyshev there's a task for upgrading to 2.15 here: [18:40:29] https://phabricator.wikimedia.org/T177201 [18:46:14] SMalyshev for point 3, i think they have added limits [18:46:27] for example topic is limited in the amount of words that is shown [18:46:36] (in a newer release) [18:48:38] 10Phabricator, 10Release-Engineering-Team (Kanban), 10Operations, 10Patch-For-Review, 10User-Elukey: Apache on phab1001 is gradually leaking worker processes which are stuck in "Gracefully finishing" state - https://phabricator.wikimedia.org/T182832#3976642 (10elukey) >>! In T182832#3976287, @mmodell wro... [18:49:35] 10Phabricator, 10Release-Engineering-Team (Kanban), 10Operations, 10Patch-For-Review, 10User-Elukey: Apache on phab1001 is gradually leaking worker processes which are stuck in "Gracefully finishing" state - https://phabricator.wikimedia.org/T182832#3976644 (10mmodell) Yeah it's especially odd that it's... [18:54:28] PROBLEM - Puppet errors on deployment-secureredirexperiment is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [19:05:32] RECOVERY - Mediawiki Error Rate on graphite-labs is OK: OK: Less than 1.00% above the threshold [1.0] [19:15:19] o/ hashar i know you're very busy! i'm just pinging you to notify you that i've revised the chromium-render CI patch according to your suggestions and i think it's ready for your review whenever you have the time: https://gerrit.wikimedia.org/r/#/c/409115/. thanks for all your help! [19:23:20] 10Phabricator, 10Release-Engineering-Team (Kanban), 10Operations, 10Patch-For-Review, 10User-Elukey: Apache on phab1001 is gradually leaking worker processes which are stuck in "Gracefully finishing" state - https://phabricator.wikimedia.org/T182832#3976742 (10mmodell) Interesting: I think I may have fo... [19:33:37] 10Phabricator, 10Release-Engineering-Team (Kanban), 10Operations, 10Patch-For-Review, 10User-Elukey: Apache on phab1001 is gradually leaking worker processes which are stuck in "Gracefully finishing" state - https://phabricator.wikimedia.org/T182832#3976774 (10Paladox) @mmodell "This is finally fixed in... [19:35:30] (03CR) 10Umherirrender: "This breaks extensions which have vendor in its phan path, because some of the dev dependency like codesniffer or php linter does not pass" [integration/config] - 10https://gerrit.wikimedia.org/r/410372 (owner: 10Legoktm) [19:38:24] 10Continuous-Integration-Infrastructure: Phan fails when vendor is part of folder list since --no-dev is removed from composer update - https://phabricator.wikimedia.org/T187489#3976784 (10Umherirrender) [19:38:36] (03CR) 10Umherirrender: "It is better to create bugs than just a comment, so T187489" [integration/config] - 10https://gerrit.wikimedia.org/r/410372 (owner: 10Legoktm) [19:40:14] 10Beta-Cluster-Infrastructure, 10MediaWiki-extensions-GlobalPreferences, 10Community-Tech-Sprint, 10Patch-For-Review: Deploy GlobalPreferences on beta cluster - https://phabricator.wikimedia.org/T184668#3976808 (10MaxSem) 05Open>03Resolved [19:41:38] 10Phabricator, 10Zero, 10MW-1.31-release-notes (WMF-deploy-2018-02-06 (1.31.0-wmf.20)): Phab file uploads are blocked for inactive Zero IP ranges - https://phabricator.wikimedia.org/T173537#3976825 (10Zoranzoki21) >>! In T173537#3954742, @Mholloway wrote: > The change is now live on ZeroWiki. I confirmed th... [19:55:56] niedzielski: yeah i noticed :) [19:56:41] Ok thanks! [19:57:14] niedzielski: I will not be able to review it tonight, but I guess I can tomorrow [19:57:17] oh [19:58:00] Ok cool. Tomorrow or Monday would be great [19:58:44] niedzielski: hmm [19:58:49] niedzielski: https://gerrit.wikimedia.org/r/#/c/409115/7/jjb/mediawiki-services.yaml that new job there [19:58:55] is the only difference to inject the env variable? [19:59:26] (03CR) 10Hashar: Update chromium-render to use Debian Chromium (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/409115 (https://phabricator.wikimedia.org/T179552) (owner: 10Niedzielski) [19:59:27] yes (i didn't know another way :|) [19:59:54] yeah that is a huge gotcha, I have exploded on that one a few weeks ago :( [20:00:04] so in short [20:00:38] the parameter should be injected by Zuul itself via the functions in zuul/parameter_functions.py [20:00:43] then we can use the regular job [20:00:45] 10Release-Engineering-Team (Kanban), 10Release, 10Train Deployments: 1.31.0-wmf.21 deployment blockers - https://phabricator.wikimedia.org/T183960#3976852 (10mmodell) [20:00:50] and keep it in experimental pipeline for now [20:00:50] grr, so when i do a manual "rebuild last build", that works because it doesn't use zuul? [20:01:21] hmm [20:01:38] so if you use that rebuild button that is a jenkins building taking care of listing the parameters from the last builda [20:01:45] and supposedly it reinjects all of them (hopefully) [20:01:57] e.g., https://integration.wikimedia.org/ci/job/chromium-render-npm-browser-node-6-docker/52/parameters/ [20:01:58] assuming the build you are rebuilding has been triggered by Zuul, it would have all the proper parameter [20:02:04] but honestly that is a bit of a mess :\ [20:02:15] yeah so in that list [20:02:23] BASE_LOG_PATH is generated / injected by zuul [20:02:36] hopefully if you rebuild that, the parameter should still be there [20:02:47] ah yeah [20:02:54] that is a rebuild already. so yeah looks like it works [20:02:57] 10Continuous-Integration-Infrastructure: Phan fails when vendor is part of folder list since --no-dev is removed from composer update - https://phabricator.wikimedia.org/T187489#3976859 (10Legoktm) a:03Legoktm I don't understand why this only failed for CiteThisPage and not every other phan extension. [20:03:37] so, on a "check experimental" build, i'm not seeing the PUPPETEER_SKIP_CHROMIUM_DOWNLOAD param injected but i'm not sure if that's because i've removed it as an experimental job or some weird issue [20:04:11] in that case, the list of parameters from the jobs are ignored entirely [20:04:22] Zuul emit a list of parameter to the Jenkins Gearman plugin [20:04:37] and that plugin does not merge the Zuul parameters with the ones defined by the jenkins job [20:04:57] since Zuul does not inject PUPPETEER_SKIP_CHROMIUM_DOWNLOAD , the build ends up running without the parameter [20:05:24] would be nicer to have the env variables defined in JJB itself, but I havent looked at thta yet [20:05:41] 10Release-Engineering-Team (Kanban), 10Release, 10Train Deployments: 1.31.0-wmf.22 deployment blockers - https://phabricator.wikimedia.org/T183961#3976870 (10mmodell) [20:05:44] 10Release-Engineering-Team (Kanban), 10Release, 10Train Deployments: 1.31.0-wmf.21 deployment blockers - https://phabricator.wikimedia.org/T183960#3976871 (10mmodell) [20:05:59] niedzielski: I am gonna hack your change and add the parameter injection [20:06:12] that is done via zuul/parameter_functions.py [20:07:47] hashar: (sorry! i'm trying to follow!) i was using apps-android-wikipedia-periodic-test as an example but i guess that's not affected by zuul? [20:08:09] yeah that job is triggered by jenkins itself [20:08:18] so it properly uses the parameters defined in the job [20:08:20] apps-android-wikipedia-periodic-test defines DISPLAY, QEMU_AUDIO_DRV [20:08:39] ohhh i see [20:13:11] (03PS8) 10Hashar: chromium-render must not download Chromium [integration/config] - 10https://gerrit.wikimedia.org/r/409115 (https://phabricator.wikimedia.org/T179552) (owner: 10Niedzielski) [20:13:30] niedzielski: https://gerrit.wikimedia.org/r/#/c/409115/8/zuul/parameter_functions.py in theory that should do it [20:13:57] oh geez [20:14:07] this way Zuul will inject the parameter for any job on mediawiki/services/chromium-render and mediawiki/services/chromium-render/deploy [20:14:30] I kept it in the experimental pipeline and if that works I guess the job can be promoted to test / gate-and-submit [20:14:31] well, that's fine with me if it's fine with you [20:14:34] 10Release-Engineering-Team (Kanban), 10Release, 10Train Deployments: 1.31.0-wmf.21 deployment blockers - https://phabricator.wikimedia.org/T183960#3976878 (10mmodell) [20:14:37] (03CR) 10Hashar: [C: 032] chromium-render must not download Chromium [integration/config] - 10https://gerrit.wikimedia.org/r/409115 (https://phabricator.wikimedia.org/T179552) (owner: 10Niedzielski) [20:14:42] that is a bit messy really :( [20:14:56] yeah :[ [20:15:21] I found some upstream bug about support an env variable to prevent downloading chromium [20:15:40] but that got rejected cause in theory one could have several version of puppetteer each requiring a specific chromium version [20:15:46] (03Merged) 10jenkins-bot: chromium-render must not download Chromium [integration/config] - 10https://gerrit.wikimedia.org/r/409115 (https://phabricator.wikimedia.org/T179552) (owner: 10Niedzielski) [20:16:16] niedzielski: deployed [20:16:55] hooray! [20:16:58] I have rechecked your patch https://gerrit.wikimedia.org/r/#/c/410621/ [20:17:14] i see! [20:17:17] and it has the parameter https://integration.wikimedia.org/ci/job/chromium-render-npm-browser-node-6-docker/53/parameters/ apparently [20:17:52] 00:00:47.495 html2pdf [20:17:52] 00:00:49.587 βœ“ should return a letter-sized PDF (2092ms) [20:17:53] !!!!!!!!!!!!!!!!!!!! [20:18:02] yep and the log show it skipped the download [20:18:17] 20:17:30 **INFO** Skipping Chromium download. "PUPPETEER_SKIP_CHROMIUM_DOWNLOAD" environment variable was found. [20:18:30] oh man that is awesome [20:19:04] πŸ‘ [20:19:31] trying on a dummy change ( https://gerrit.wikimedia.org/r/#/c/399587/ ) https://integration.wikimedia.org/ci/job/chromium-render-npm-browser-node-6-docker/54/console [20:21:12] ok i think those failures are because it doesn't have CHROME_BIN [20:21:12] 10Continuous-Integration-Infrastructure: Phan fails when vendor is part of folder list since --no-dev is removed from composer update - https://phabricator.wikimedia.org/T187489#3976884 (10Legoktm) Oh, I see. For example Cite only lints the includes/ directory, so vendor/ won't get picked up. CiteThisPage lints... [20:22:02] (there's a change in the chromium-render repo that's needed, the one we checked earlier, so it'd need to be rebased onto that patchset) [20:22:25] niedzielski: I guess that is https://gerrit.wikimedia.org/r/#/c/410355/ ? [20:22:43] I commented on that one, there are a bunch of dependencies you can remove since you are adding chromium.deb [20:22:52] https://gerrit.wikimedia.org/r/#/c/398529/ is a cleanup I did a few months ago [20:23:07] tldr: since you are adding chromium.deb , almost all the other dependencies can be removed [20:23:11] hashar: sorry, this guy: https://gerrit.wikimedia.org/r/#/c/410621/ [20:23:24] oh [20:24:20] 10Release-Engineering-Team (Kanban), 10Release, 10Train Deployments: 1.31.0-wmf.21 deployment blockers - https://phabricator.wikimedia.org/T183960#3976894 (10mmodell) [20:24:30] hashar: ah, sorry. i'm still trying to get up to speed on this crazy project. thank you for patch! i'll check that out and try to get it in [20:26:58] niedzielski: and probably you could squash both of your patches into one: https://gerrit.wikimedia.org/r/#/c/410355/ https://gerrit.wikimedia.org/r/#/c/410621/ [20:27:02] but I don't know really [20:27:23] once you are happy with them, we can promote the job in CI (anyone from releng + a few others can deploy it) [20:27:31] then +2 your change and they should get merged [20:27:43] (or just force merge them and we deploy the CI patch after) [20:28:37] (following up in gerrit) [20:29:56] Yippee, build fixed! [20:29:56] Project selenium-Wikibase-chrome Β» chrome,beta,Linux,DebianJessie && contintLabsSlave build #111: 09FIXED in 42 min: https://integration.wikimedia.org/ci/job/selenium-Wikibase-chrome/BROWSER=chrome,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=DebianJessie%20&&%20contintLabsSlave/111/ [20:29:56] 10Continuous-Integration-Config, 10Security-Team, 10phan-taint-check-plugin, 10MediaWiki-Platform-Team (MWPT-Q3-Jan-Mar-2018), 10Patch-For-Review: Make jenkins run phan-taint-check-plugin non-voting and then voting - https://phabricator.wikimedia.org/T182599#3976964 (10Legoktm) [20:30:50] 10Continuous-Integration-Infrastructure (shipyard), 10Release-Engineering-Team (Kanban), 10Patch-For-Review: Migrate operations-mw-config-composer-hhvm-jessie to Docker - https://phabricator.wikimedia.org/T186145#3976969 (10hashar) With the containers of https://gerrit.wikimedia.org/r/#/c/410990/ I got the t... [20:31:08] i've kept the patches separate because one is more releng / CI-centric and the other is more services-centric [20:31:23] (they are closely related though!) [20:31:49] (03CR) 10Hashar: [C: 032] "I need the new containers and will craft the job for operations/mediawiki-config in https://gerrit.wikimedia.org/r/#/c/410950/" [integration/config] - 10https://gerrit.wikimedia.org/r/410990 (https://phabricator.wikimedia.org/T186145) (owner: 10Hashar) [20:32:03] niedzielski: makes sense [20:32:28] niedzielski: one thing to remember is that apparently if one use the puppeter_skip_download, executablePath is still set to some arbitrary place [20:32:36] so it has to be set explicitly to /usr/bin/chromium [20:32:44] and i am not sure how service-runner deal with that [20:32:55] (03Merged) 10jenkins-bot: docker: script to only install composer dev dependencies [integration/config] - 10https://gerrit.wikimedia.org/r/410990 (https://phabricator.wikimedia.org/T186145) (owner: 10Hashar) [20:33:43] hashar: if CHROME_BIN was undefined, this would cause problems. however, it's set as a deploy (production) environment variable so i think it's fine [20:34:20] !log Updating docker-pkg files on contint1001 for https://gerrit.wikimedia.org/r/#/c/410990/ (no jobs touched) [20:34:26] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [20:34:28] niedzielski: \o/ [20:35:00] (03CR) 10Hashar: [C: 04-1] "With the containers of https://gerrit.wikimedia.org/r/#/c/410990/ I got the test suite to run properly with:" [integration/config] - 10https://gerrit.wikimedia.org/r/410950 (https://phabricator.wikimedia.org/T186145) (owner: 10Hashar) [20:35:22] hashar: ok, well i will ping my colleagues to try and get that CHROME_BIN patch pushed through. then i'll submit a promotion patch for the job and ping a non-hashar releng. thank you so so much for all your help! [20:35:49] niedzielski: awesome!!! [20:36:01] niedzielski: and thank you for all the care to use the Debian Chromium package :] [20:36:16] (which might well later cause a bit of a mess when chromium get upgraded) [20:36:40] πŸ‘πŸ‘πŸ‘ [20:36:42] time will tell! [20:43:32] niedzielski: hmm https://gerrit.wikimedia.org/r/#/c/410621/ got a +1 :] [20:43:40] so maybe promote the CI job and then +2 the change? [20:44:20] will do! [20:48:48] (03PS1) 10Niedzielski: Update: promote chromium-render to test and gate-and-submit [integration/config] - 10https://gerrit.wikimedia.org/r/411085 (https://phabricator.wikimedia.org/T179552) [20:50:00] (03CR) 10Hashar: [C: 032] "Lets fly since the experimental job works!" [integration/config] - 10https://gerrit.wikimedia.org/r/411085 (https://phabricator.wikimedia.org/T179552) (owner: 10Niedzielski) [20:51:14] (03Merged) 10jenkins-bot: Update: promote chromium-render to test and gate-and-submit [integration/config] - 10https://gerrit.wikimedia.org/r/411085 (https://phabricator.wikimedia.org/T179552) (owner: 10Niedzielski) [20:52:50] ok download skipped. that's good [20:53:02] nice [20:53:07] !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! [20:56:13] hashar: thanks again for all the help! piotr just stepped out but i'm sure he'll +2 the remaining patch when he gets back [20:56:25] niedzielski: awesome! [20:56:39] niedzielski: please make sure to share the good news with whoever is in charge of chromium-render :] [20:57:33] hashar: well, that's currently the web folks but i believe it will be transitioning to ops next quarter? the project is very hazy :D [20:57:47] i will follow up on the ticket so there's no confusion over the current status [20:57:49] thank you! [20:58:34] niedzielski: you are welcome. I am quite happy to see that one coming to an end \o/ [20:58:57] :D [21:00:07] 10Continuous-Integration-Config, 10Release-Engineering-Team (Kanban), 10Proton, 10Readers-Web-Backlog, and 2 others: Set up Jenkins for chromium-render repository - https://phabricator.wikimedia.org/T179552#3977148 (10Niedzielski) Thanks @hashar for getting those last couple patches across the finish line!... [21:00:34] 10Continuous-Integration-Infrastructure, 10MediaWiki-extensions-General, 10MW-1.31-release-notes (WMF-deploy-2018-02-20 (1.31.0-wmf.22)), 10Patch-For-Review: Create composer package that contains most of the MediaWiki extension phan config instead of copy/p... - https://phabricator.wikimedia.org/T186315#3940690 [21:06:25] 10Continuous-Integration-Config, 10Release-Engineering-Team (Kanban), 10Proton, 10Readers-Web-Backlog, and 2 others: Set up Jenkins for chromium-render repository - https://phabricator.wikimedia.org/T179552#3977162 (10Niedzielski) a:05Niedzielski>03pmiazga [21:08:17] SMalyshev https://bugs.chromium.org/p/gerrit/issues/detail?id=4552 [21:08:22] PROBLEM - Puppet errors on deployment-mx02 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [21:08:37] SMalyshev https://gerrit-review.googlesource.com/#/c/gerrit/+/159890/ [21:10:54] paladox: thanks, looks like people are working on this [21:11:01] yep. [21:11:10] upstream are busy improving based on feddback [21:11:19] the more you give, the more they can prioritse things [21:13:59] (03PS1) 10Hashar: Add support to pass options to 'docker run' [integration/config] - 10https://gerrit.wikimedia.org/r/411098 (https://phabricator.wikimedia.org/T186145) [21:26:06] paladox: is polygerrit just some html/javascript? [21:26:12] hashar yep [21:26:14] no java [21:26:22] hashar very easy to edit :) [21:26:36] it uses soy for it's index.html [21:26:42] so that we get the baseUrl [21:26:50] needed for it to work on /r or any other base urls [21:26:54] paladox: one thing that would be quite awesome is to write a step by step tutorial to hack on polygerrit and maybe some other from our community will join the fun [21:26:59] and make gerrit ui nicer :] [21:27:09] hashar heh [21:27:22] polygerrit plugins are supported from 2.15+ [21:27:33] ;] [21:28:05] hashar i wrote this https://gerrit-review.googlesource.com/#/c/plugins/delete-project/+/140591/ (based on chromium polygerrit plugin) [21:28:55] Gerrit.install(function(plugin) { [21:28:55] plugin.registerCustomComponent( [21:28:55] 'repo-command', 'repo-command-delete-repo'); [21:28:55] }); [21:29:19] hashar gerrit uses the term "Repo or repository now" [21:29:29] (03PS3) 10Hashar: Migrate mw-config composer job to Docker [integration/config] - 10https://gerrit.wikimedia.org/r/410950 (https://phabricator.wikimedia.org/T186145) [21:29:54] 10Phabricator, 10Release-Engineering-Team (Kanban), 10Operations, 10Patch-For-Review, 10User-Elukey: Apache on phab1001 is gradually leaking worker processes which are stuck in "Gracefully finishing" state - https://phabricator.wikimedia.org/T182832#3977197 (10Dzahn) >>! In T182832#3974829, @elukey wrote... [21:30:11] 10Continuous-Integration-Infrastructure (shipyard), 10Release-Engineering-Team (Kanban), 10Patch-For-Review: Migrate operations-mw-config-composer-hhvm-jessie to Docker - https://phabricator.wikimedia.org/T186145#3977202 (10hashar) I forged a job that seems to pass all fine and do exactly what we want: https... [21:30:48] PROBLEM - Free space - all mounts on deployment-mediawiki04 is CRITICAL: CRITICAL: deployment-prep.deployment-mediawiki04.diskspace.root.byte_percentfree (<11.11%) [21:31:37] (03CR) 10Hashar: "My use case is to reuse the releng/composer-hhvm container but with a different entrypoint via:" [integration/config] - 10https://gerrit.wikimedia.org/r/411098 (https://phabricator.wikimedia.org/T186145) (owner: 10Hashar) [21:31:38] hashar upstream are now prepairing to send out a annoucement to discourage the use of the old ui on gerrit-review. They consider polygerrit to now be feature parity with gwtui. [21:31:40] i didn't realize for a long time that i can also change the theme for the inline editor.. and how many themes you can select from now [21:31:49] mutante yep codemirror :) [21:31:56] it looks alot prettier in polygerrit [21:31:58] yea [21:32:15] but that was me testing, they havent added themes yet in polygerrit for inline editing. [21:32:30] oh,ok [21:32:47] (03CR) 10Hashar: "Seems to work fine https://integration.wikimedia.org/ci/job/operations-mw-config-composer-test-docker/" [integration/config] - 10https://gerrit.wikimedia.org/r/410950 (https://phabricator.wikimedia.org/T186145) (owner: 10Hashar) [21:33:16] mutante i was the one that fixed syntax highlighting in codemirror-editor + also fixed the support for preferences :) [21:33:34] paladox: in gerrit 2.14 I am missing a few links here and there. But maybe I am not used to polygerrit yet [21:33:37] I should give it another try [21:33:49] hashar which links? [21:34:24] 10Continuous-Integration-Infrastructure, 10MW-1.31-release-notes (WMF-deploy-2018-02-20 (1.31.0-wmf.22)), 10Patch-For-Review: Phan fails when vendor is part of folder list since --no-dev is removed from composer update - https://phabricator.wikimedia.org/T187489#3977227 (10Legoktm) Theoretically this affects... [21:37:32] hashar this is the new dashboard https://gerrit-review.googlesource.com/q/status:open?polygerrit=1 which is currently being redesgn again [21:37:47] https://bugs.chromium.org/p/gerrit/issues/detail?id=8362 [21:38:35] PROBLEM - Puppet errors on saucelabs-01 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [21:39:46] paladox: nice:) [21:39:58] mutante someone called that an excel sheet (lol) [22:00:28] it does have a spreadsheetish appearance [22:00:51] heh [22:03:57] Project mwext-phpunit-coverage-publish build #1032: 04FAILURE in 35 sec: https://integration.wikimedia.org/ci/job/mwext-phpunit-coverage-publish/1032/ [22:04:22] Yippee, build fixed! [22:04:23] Project mwext-phpunit-coverage-publish build #1033: 09FIXED in 25 sec: https://integration.wikimedia.org/ci/job/mwext-phpunit-coverage-publish/1033/ [22:15:13] 10Continuous-Integration-Infrastructure, 10Patch-For-Review, 10Release Pipeline (Blubber): jenkins-slave@contint1001 not a member of docker group (CI tests for mathoid broken) - https://phabricator.wikimedia.org/T186790#3977282 (10thcipriani) >>! In T186790#3975275, @akosiaris wrote: > It already has been. j... [22:16:21] 10Phabricator, 10Zero, 10MW-1.31-release-notes (WMF-deploy-2018-02-06 (1.31.0-wmf.20)): Phab file uploads are blocked for inactive Zero IP ranges - https://phabricator.wikimedia.org/T173537#3977284 (10Zoranzoki21) Suggestion: Whitelist my IP range 109.245.0.0/16 [22:18:34] RECOVERY - Puppet errors on saucelabs-01 is OK: OK: Less than 1.00% above the threshold [0.0] [22:25:42] 10Phabricator, 10Zero, 10MW-1.31-release-notes (WMF-deploy-2018-02-06 (1.31.0-wmf.20)): Phab file uploads are blocked for inactive Zero IP ranges - https://phabricator.wikimedia.org/T173537#3977291 (10Mholloway) >>! In T173537#3977284, @Zoranzoki21 wrote: > Suggestion: Whitelist my IP range 109.245.0.0/16 T... [22:27:08] 10Phabricator, 10Zero, 10MW-1.31-release-notes (WMF-deploy-2018-02-06 (1.31.0-wmf.20)): Phab file uploads are blocked for inactive Zero IP ranges - https://phabricator.wikimedia.org/T173537#3977302 (10Zoranzoki21) >>! In T173537#3977291, @Mholloway wrote: >>>! In T173537#3977284, @Zoranzoki21 wrote: >> Sugge... [22:44:09] 10Continuous-Integration-Infrastructure, 10Patch-For-Review, 10Release Pipeline (Blubber): jenkins-slave@contint1001 not a member of docker group (CI tests for mathoid broken) - https://phabricator.wikimedia.org/T186790#3977351 (10hashar) Thank you for the hotfix @akosiaris !!! I guess the long term fix i... [22:47:35] 10Continuous-Integration-Infrastructure (shipyard), 10Release-Engineering-Team (Kanban), 10Patch-For-Review: Migrate operations-mw-config-composer-hhvm-jessie to Docker - https://phabricator.wikimedia.org/T186145#3977371 (10hashar) >>! In T186145#3939064, @Addshore wrote: >>>! In T186145#3938606, @hashar wro... [22:47:59] hashar which links are missing? :) [22:48:25] paladox: I will switch to polygerrit and take notes of annoyances as I encounter them :) [22:48:31] most probably they are all already fixed [22:48:31] ok [22:48:38] dont worry! [22:48:49] hashar the admin links in 2.14 will head to a page saying going to old ui [22:48:52] I guess I am not in the mood to change the interface yet [22:48:56] that's at least fixed in 2.15. [22:49:46] hashar i seem to use polygerrit on gerrit.wikimedia.org as it's fast :) [22:52:14] ;) [22:52:21] I am off to bed. Have a good night! [22:52:40] Bonne nuit [22:53:08] What's Bonne Nuit? [22:53:10] Hauskatze ^^ [22:53:15] is that good night in french? [22:53:30] paladox: yes [22:56:21] ok [23:13:49] 10Phabricator, 10Release-Engineering-Team (Kanban), 10Operations, 10Patch-For-Review, 10User-Elukey: Apache on phab1001 is gradually leaking worker processes which are stuck in "Gracefully finishing" state - https://phabricator.wikimedia.org/T182832#3977421 (10Paladox) @Muehlenhoff hi, what about backpor... [23:30:21] !log Ran cleanupSpam.php on deploymentwiki to get rid of a bunch of crap. [23:30:27] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [23:51:51] PROBLEM - Free space - all mounts on deployment-eventlog02 is CRITICAL: CRITICAL: deployment-prep.deployment-eventlog02.diskspace.root.byte_percentfree (<30.00%)