[00:13:49] I'm sure I've asked this before but don't remember the answer... is there any way at all to see/debug php unit test on Jenkins? [00:14:16] i.e. get the exact VM it is running in, log in there and check what's going on? [00:22:53] 10Continuous-Integration-Config, 10Operations: add ci test for admin module indentation - https://phabricator.wikimedia.org/T190766#4083108 (10RobH) p:05Triage>03Normal [00:31:15] PROBLEM - Long lived cherry-picks on puppetmaster on deployment-puppetmaster02 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [00:34:09] SMalyshev: if you click on the name of a test in the comment from jenkins-bot on gerrit.. and get to the output page.. at the top of it it mentions the name of the VM [00:34:23] f.e. https://integration.wikimedia.org/ci/job/operations-puppet-tests-docker/18246/console says Building remotely on integration-slave-docker-1012 [00:34:38] integration-slave-docker-1012 is the instance name to connect to [00:34:47] dunno about permissions for the project though.. [00:35:37] mutante: yes, thanks, I get the hostname but looks like I'm not allowed to log in [00:39:50] SMalyshev: you should probably ask hashar if you can be added to it [00:40:02] it = the right project in Horizon [00:40:22] mutante: ok, thanks, will try to find hashar [00:41:22] it's some crazy situation where one of the global config vars get set to "' (literally, two quotes) and I can't find who'd doing it or why [00:50:02] oh duh... found it: $wgLexemeLanguageCodePropertyId = $evilConfig;. That's not how things should be done in unit tests :( [01:36:45] 10Phabricator, 10Discourse, 10Developer-Relations (Jan-Mar-2018): Enable Wikimedia Phabricator login in discourse-mediawiki.wmflabs.org - https://phabricator.wikimedia.org/T184987#4083260 (10mmodell) I've added the repo to phabricator: {rDISCPHAB} [01:40:38] PROBLEM - Free space - all mounts on deployment-fluorine02 is CRITICAL: CRITICAL: deployment-prep.deployment-fluorine02.diskspace._srv.byte_percentfree (<44.44%) [03:00:10] Project mediawiki-core-code-coverage build #3407: 04STILL FAILING in 9.3 sec: https://integration.wikimedia.org/ci/job/mediawiki-core-code-coverage/3407/ [03:00:10] Project mediawiki-core-code-coverage-php7 build #168: 04STILL FAILING in 9.4 sec: https://integration.wikimedia.org/ci/job/mediawiki-core-code-coverage-php7/168/ [03:07:20] uhoh [03:07:31] 03:00:07 DEBUG:git.cmd:AutoInterrupt wait stderr: 'fatal: protocol error: bad pack header' [03:07:56] no_justification: paladox: I guess this is the git remote prune issue? [03:10:11] !log deleted mediawiki-core-code-coverage* workspaces to work around git/gerrit issue and retriggered jobs [03:10:12] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [03:10:28] both are running now [04:08:27] Yippee, build fixed! [04:08:28] Project mediawiki-core-code-coverage-php7 build #169: 09FIXED in 58 min: https://integration.wikimedia.org/ci/job/mediawiki-core-code-coverage-php7/169/ [05:23:48] Yippee, build fixed! [05:23:49] Project mediawiki-core-code-coverage build #3408: 09FIXED in 2 hr 15 min: https://integration.wikimedia.org/ci/job/mediawiki-core-code-coverage/3408/ [06:55:39] RECOVERY - Free space - all mounts on deployment-fluorine02 is OK: OK: All targets OK [07:02:01] legoktm: yep [07:24:24] (03CR) 10Thiemo Kreuz (WMDE): [C: 032] "I did a quick profile run and oh boy this is sooo worth it! The tokenIsNamespaced function accounts for almost 100% of the total runtime o" (031 comment) [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/421954 (owner: 10Umherirrender) [07:25:10] (03Merged) 10jenkins-bot: Improve performance of PrefixedGlobalFunctionsSniff [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/421954 (owner: 10Umherirrender) [07:25:42] (03CR) 10jenkins-bot: Improve performance of PrefixedGlobalFunctionsSniff [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/421954 (owner: 10Umherirrender) [07:52:44] 10Beta-Cluster-Infrastructure: Secure deployment-prep sudo access to prevent member -> admin escalation by dns-manager credentials - https://phabricator.wikimedia.org/T190781#4083552 (10Krenair) [07:53:27] 10Beta-Cluster-Infrastructure: Secure deployment-prep sudo access to prevent member -> projectadmin escalation by dns-manager credentials - https://phabricator.wikimedia.org/T190781#4083552 (10Krenair) [08:30:31] !log Update cxserver to 9e8ebda [08:30:33] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [09:52:14] (03PS2) 10Hashar: (WIP) Docker container with Stretch [integration/quibble] - 10https://gerrit.wikimedia.org/r/421900 [09:52:16] (03PS1) 10Hashar: Ensure there is a `log` dir in the workspace [integration/quibble] - 10https://gerrit.wikimedia.org/r/422107 [09:52:18] (03PS1) 10Hashar: Port mw-fetch-composer-dev.sh to Python [integration/quibble] - 10https://gerrit.wikimedia.org/r/422108 [09:52:20] (03PS1) 10Hashar: Run QUnit headlessly when there is no DISPLAY [integration/quibble] - 10https://gerrit.wikimedia.org/r/422109 (https://phabricator.wikimedia.org/T167507) [09:52:47] (03CR) 10Hashar: [C: 032] Ensure there is a `log` dir in the workspace [integration/quibble] - 10https://gerrit.wikimedia.org/r/422107 (owner: 10Hashar) [09:52:53] (03CR) 10Hashar: [C: 032] Port mw-fetch-composer-dev.sh to Python [integration/quibble] - 10https://gerrit.wikimedia.org/r/422108 (owner: 10Hashar) [09:53:11] (03Merged) 10jenkins-bot: Ensure there is a `log` dir in the workspace [integration/quibble] - 10https://gerrit.wikimedia.org/r/422107 (owner: 10Hashar) [09:53:17] (03Merged) 10jenkins-bot: Port mw-fetch-composer-dev.sh to Python [integration/quibble] - 10https://gerrit.wikimedia.org/r/422108 (owner: 10Hashar) [10:20:02] 10Phabricator: Make task ids easily copyable in workboard - https://phabricator.wikimedia.org/T190793#4083858 (10Sebastian_Berlin-WMSE) [10:22:58] 10Phabricator, 10WMSE-Bug-Reporting-and-Translation-2018: Make task ids easily copyable in workboard - https://phabricator.wikimedia.org/T190793#4083894 (10Sebastian_Berlin-WMSE) [10:54:07] PROBLEM - Host deployment-videoscaler01 is DOWN: CRITICAL - Host Unreachable (10.68.19.130) [10:54:52] PROBLEM - Host deployment-tmh01 is DOWN: CRITICAL - Host Unreachable (10.68.16.211) [10:58:46] 10Gerrit, 10Operations: git pull fails for MW core with "fatal: protocol error: bad pack header" when local branches point to remote branches that no more exist - https://phabricator.wikimedia.org/T190676#4084023 (10Ladsgroup) 05Resolved>03Open We can't tell anyone to run "git remote prune origin" when see... [11:27:07] 10Beta-Cluster-Infrastructure, 10Puppet: Puppet broken on deployment-ms-be03 - https://phabricator.wikimedia.org/T190683#4084098 (10MarcoAurelio) @Joe Thanks for having a look. I however don't know what to do to fix this. Any tips? Thanks! [11:33:22] 10Gerrit, 10Operations: git pull fails for MW core with "fatal: protocol error: bad pack header" when local branches point to remote branches that no more exist - https://phabricator.wikimedia.org/T190676#4084113 (10Aklapper) I propose to decline this task as this is an upstream issue. [11:39:49] 10Phabricator, 10WMSE-Bug-Reporting-and-Translation-2018: Make task ids easily copyable in workboard - https://phabricator.wikimedia.org/T190793#4084120 (10Aklapper) Why do you need to copy task IDs from the workboard? What exactly happens after copying them? [11:43:08] 10Diffusion: rEBSP in "importing" status since August 2016 - https://phabricator.wikimedia.org/T190802#4084124 (10MarcoAurelio) [11:52:53] 10Gerrit, 10Operations: git pull fails for MW core with "fatal: protocol error: bad pack header" when local branches point to remote branches that no more exist - https://phabricator.wikimedia.org/T190676#4084154 (10Paladox) This has nothing to do with gerrit now, the problem lies in git not doing this behavio... [11:55:49] 10Phabricator, 10WMSE-Bug-Reporting-and-Translation-2018: Make task ids easily copyable in workboard - https://phabricator.wikimedia.org/T190793#4084175 (10Sebastian_Berlin-WMSE) >>! In T190793#4084120, @Aklapper wrote: > Why do you need to copy task IDs from the workboard? For various reasons. Often it's to a... [12:10:03] 10Release-Engineering-Team (Kanban), 10MW-1.31-release-notes (WMF-deploy-2018-03-27 (1.31.0-wmf.27)), 10Patch-For-Review, 10User-zeljkofilipin, 10Wikimedia-log-errors (Jenkins Failure): Warning: Task "stylelint:src" failed due to postcss-less@1.1.4 - https://phabricator.wikimedia.org/T190269#4084217 (10ze... [12:16:03] (03PS3) 10Hashar: (WIP) Docker container with Stretch [integration/quibble] - 10https://gerrit.wikimedia.org/r/421900 [12:16:05] (03PS1) 10Hashar: Disable Chromium sandbox when running in a container [integration/quibble] - 10https://gerrit.wikimedia.org/r/422130 [12:45:20] 10Phabricator (Upstream), 10WMSE-Bug-Reporting-and-Translation-2018, 10Upstream: Make task ids easily copyable in workboard - https://phabricator.wikimedia.org/T190793#4084283 (10Aklapper) p:05Triage>03Low [13:01:28] 10Gerrit, 10Operations: git pull fails for MW core with "fatal: protocol error: bad pack header" when local branches point to remote branches that no more exist - https://phabricator.wikimedia.org/T190676#4084291 (10daniel) @Ladsgroup I agree that it's a bug and should be fixed. But if I understand correctly,... [13:28:20] (03PS1) 10Mholloway: Whitelist Jberkel [integration/config] - 10https://gerrit.wikimedia.org/r/422147 [13:28:36] (03CR) 10Hashar: [C: 032] "karma:main works on a host machine as well as in a Docker container." [integration/quibble] - 10https://gerrit.wikimedia.org/r/422109 (https://phabricator.wikimedia.org/T167507) (owner: 10Hashar) [13:29:02] (03Merged) 10jenkins-bot: Run QUnit headlessly when there is no DISPLAY [integration/quibble] - 10https://gerrit.wikimedia.org/r/422109 (https://phabricator.wikimedia.org/T167507) (owner: 10Hashar) [13:30:28] 10Gerrit: Install motd plugin - https://phabricator.wikimedia.org/T190810#4084375 (10Paladox) [13:34:47] (03CR) 10Hashar: [C: 032] "Good for Docker. webdriver/chromedriver has a similar issue, that would be solved in mediawiki/core" [integration/quibble] - 10https://gerrit.wikimedia.org/r/422130 (owner: 10Hashar) [13:35:12] (03Merged) 10jenkins-bot: Disable Chromium sandbox when running in a container [integration/quibble] - 10https://gerrit.wikimedia.org/r/422130 (owner: 10Hashar) [13:37:02] (03PS2) 10Hashar: Split PHPUnit Database tests [integration/quibble] - 10https://gerrit.wikimedia.org/r/421866 (https://phabricator.wikimedia.org/T190554) [13:38:03] (03PS3) 10Hashar: Split PHPUnit Database tests [integration/quibble] - 10https://gerrit.wikimedia.org/r/421866 (https://phabricator.wikimedia.org/T190554) [13:39:41] (03CR) 10Mholloway: Add mobileapps-diff-test periodic job (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/415886 (https://phabricator.wikimedia.org/T177896) (owner: 10Mholloway) [13:54:09] (03CR) 10Hashar: [C: 032] "mediawiki/core fix https://gerrit.wikimedia.org/r/#/c/421789/ got merged." [integration/quibble] - 10https://gerrit.wikimedia.org/r/421866 (https://phabricator.wikimedia.org/T190554) (owner: 10Hashar) [13:54:33] (03Merged) 10jenkins-bot: Split PHPUnit Database tests [integration/quibble] - 10https://gerrit.wikimedia.org/r/421866 (https://phabricator.wikimedia.org/T190554) (owner: 10Hashar) [14:23:14] no_justification looks like upstream are dropping groups from reviewdb now https://gerrit-review.googlesource.com/c/gerrit/+/168533 [14:53:16] PROBLEM - Host deployment-puppetdb01 is DOWN: CRITICAL - Host Unreachable (10.68.23.76) [14:54:49] PROBLEM - Puppet errors on deployment-zookeeper02 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [14:58:32] PROBLEM - Puppet errors on deployment-maps03 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [15:07:27] (03PS1) 10Vgutierrez: Upgrade mtail in operations-puppet [integration/config] - 10https://gerrit.wikimedia.org/r/422168 (https://phabricator.wikimedia.org/T184942) [15:13:33] RECOVERY - Puppet errors on deployment-maps03 is OK: OK: Less than 1.00% above the threshold [0.0] [15:17:52] (03PS1) 10Vgutierrez: Docker: use new operations-puppet image [integration/config] - 10https://gerrit.wikimedia.org/r/422169 (https://phabricator.wikimedia.org/T184942) [15:22:09] (03CR) 10Hashar: [C: 032] Upgrade mtail in operations-puppet [integration/config] - 10https://gerrit.wikimedia.org/r/422168 (https://phabricator.wikimedia.org/T184942) (owner: 10Vgutierrez) [15:23:23] (03Merged) 10jenkins-bot: Upgrade mtail in operations-puppet [integration/config] - 10https://gerrit.wikimedia.org/r/422168 (https://phabricator.wikimedia.org/T184942) (owner: 10Vgutierrez) [15:23:38] !log building docker-registry.wikimedia.org/releng/operations-puppet:0.3.1 | https://gerrit.wikimedia.org/r/#/c/422168/ [15:23:40] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [15:29:48] RECOVERY - Puppet errors on deployment-zookeeper02 is OK: OK: Less than 1.00% above the threshold [0.0] [15:35:14] 10Beta-Cluster-Infrastructure, 10Privacy: Flush private data on Beta Cluster - https://phabricator.wikimedia.org/T189541#4084944 (10Tgr) Plan: * log in to `deployment-fluorine02.eqiad.wmflabs`, run `sudo rm -f `/srv/mw-log/*.log /srv/mw-log/archive/*` * at https://logstash-beta.wmflabs.org/app/kibana#/dev_tool... [15:37:18] 10Gerrit: Install motd plugin - https://phabricator.wikimedia.org/T190810#4084953 (10Aklapper) How would installing the motd plugin in itself magically resolve T190676? I don't get it. [15:38:05] 10Gerrit: Install motd plugin - https://phabricator.wikimedia.org/T190810#4084965 (10Paladox) it would provide the info when the user clones so that we doin't have to keep telling the user to run this command. [15:38:06] (03CR) 10Hashar: [C: 032] "Job deployed and that seems to work based on a recheck of https://gerrit.wikimedia.org/r/#/c/422155/" [integration/config] - 10https://gerrit.wikimedia.org/r/422169 (https://phabricator.wikimedia.org/T184942) (owner: 10Vgutierrez) [15:39:19] (03Merged) 10jenkins-bot: Docker: use new operations-puppet image [integration/config] - 10https://gerrit.wikimedia.org/r/422169 (https://phabricator.wikimedia.org/T184942) (owner: 10Vgutierrez) [15:47:00] 10Diffusion: rEBSP in "importing" status since August 2016 - https://phabricator.wikimedia.org/T190802#4085021 (10mmodell) 05Open>03Resolved a:03mmodell Fixed [15:50:27] 10Gerrit: Install motd plugin - https://phabricator.wikimedia.org/T190810#4085028 (10Aklapper) @Paladox: I hope I misunderstand you and you do not want to spam every single user with info about [[ https://www.mediawiki.org/wiki/Gerrit/Troubleshooting | random potential errors ]] that a user might run into once i... [15:58:54] 10Gerrit, 10Release-Engineering-Team (Kanban): Install motd plugin - https://phabricator.wikimedia.org/T190810#4085075 (10demon) p:05Triage>03Normal a:03demon I want to install this anyway to communicate downtime to users ahead of time. [16:05:35] 10Gerrit, 10Release-Engineering-Team (Kanban), 10Patch-For-Review: Install motd plugin - https://phabricator.wikimedia.org/T190810#4085130 (10Aklapper) >>! In T190810#4085075, @demon wrote: > I want to install this anyway to communicate downtime to users ahead of time. That makes sense - using it for announ... [16:25:47] 10Gerrit, 10Operations: git pull fails for MW core with "fatal: protocol error: bad pack header" when local branches point to remote branches that no more exist - https://phabricator.wikimedia.org/T190676#4085208 (10Ladsgroup) 05Open>03declined If you think so I'm fine but please document this somewhere so... [16:28:13] 10Gerrit, 10Release-Engineering-Team (Kanban), 10Patch-For-Review: Install motd plugin - https://phabricator.wikimedia.org/T190810#4085216 (10demon) It wasn't meant to be a knowledge base of possibilities. You misunderstand the premise. The relation of T190676 to this was that it's an //active// issue peopl... [16:31:52] 10Release-Engineering-Team, 10Epic, 10MW-1.31-release-notes (WMF-deploy-2018-02-27 (1.31.0-wmf.23)), 10Patch-For-Review, 10User-zeljkofilipin: Q3 Selenium framework improvements - https://phabricator.wikimedia.org/T182421#4085231 (10zeljkofilipin) [16:31:54] 10Release-Engineering-Team (Kanban), 10MediaWiki-Core-Tests, 10MW-1.31-release-notes (WMF-deploy-2018-03-27 (1.31.0-wmf.27)), 10Patch-For-Review, 10User-zeljkofilipin: Replace nodemw with mwbot - https://phabricator.wikimedia.org/T181284#4085229 (10zeljkofilipin) 05Open>03Resolved Updated documentation. [16:36:14] 10Release-Engineering-Team (Kanban), 10User-zeljkofilipin: Video recording for Selenium tests in Node.js - https://phabricator.wikimedia.org/T179188#4085248 (10zeljkofilipin) [16:51:38] Project mwext-phpunit-coverage-publish build #2638: 04FAILURE in 1 min 12 sec: https://integration.wikimedia.org/ci/job/mwext-phpunit-coverage-publish/2638/ [16:57:44] 10Gerrit, 10Release-Engineering-Team (Kanban), 10Patch-For-Review: Install motd plugin - https://phabricator.wikimedia.org/T190810#4085384 (10Aklapper) I see. I wasn't aware that several people are currently affected by it. I am sorry for my wrong understanding here, @Paladox. [17:13:51] (03PS1) 10BearND: Add link to show how to whitelist a user [integration/config] - 10https://gerrit.wikimedia.org/r/422187 [17:20:57] 10Release-Engineering-Team, 10Epic, 10Tracking, 10User-zeljkofilipin: Selenium framework improvements - https://phabricator.wikimedia.org/T182986#4085473 (10zeljkofilipin) [17:20:59] 10Release-Engineering-Team (Kanban), 10Wikimedia-Hackathon-2018, 10User-zeljkofilipin: Write Selenium tests in JavaScript/Node.js - https://phabricator.wikimedia.org/T190046#4085472 (10zeljkofilipin) [17:21:12] 10Release-Engineering-Team, 10Epic, 10Tracking, 10User-zeljkofilipin: Selenium framework improvements - https://phabricator.wikimedia.org/T182986#3840359 (10zeljkofilipin) [17:21:14] 10Release-Engineering-Team (Kanban), 10Wikimedia-Hackathon-2018, 10User-zeljkofilipin: Pair on writing Selenium tests in JavaScript/Node.js - https://phabricator.wikimedia.org/T190687#4085474 (10zeljkofilipin) [17:41:41] (03PS4) 10Hashar: (WIP) Docker container with Stretch [integration/quibble] - 10https://gerrit.wikimedia.org/r/421900 [17:41:43] (03PS1) 10Hashar: Set some useful env variables [integration/quibble] - 10https://gerrit.wikimedia.org/r/422193 [17:41:45] (03PS1) 10Hashar: Monkey patch $wgTmpDirectory [integration/quibble] - 10https://gerrit.wikimedia.org/r/422194 [17:42:39] (03PS5) 10Hashar: Docker container based on Stretch [integration/quibble] - 10https://gerrit.wikimedia.org/r/421900 [17:42:41] (03PS1) 10Hashar: (WIP) drop to bash on failure [integration/quibble] - 10https://gerrit.wikimedia.org/r/422195 [17:43:05] (03CR) 10jerkins-bot: [V: 04-1] (WIP) drop to bash on failure [integration/quibble] - 10https://gerrit.wikimedia.org/r/422195 (owner: 10Hashar) [17:44:13] (03CR) 10Hashar: [C: 032] Set some useful env variables [integration/quibble] - 10https://gerrit.wikimedia.org/r/422193 (owner: 10Hashar) [17:44:17] (03CR) 10Hashar: [C: 032] Monkey patch $wgTmpDirectory [integration/quibble] - 10https://gerrit.wikimedia.org/r/422194 (owner: 10Hashar) [17:44:37] (03Merged) 10jenkins-bot: Set some useful env variables [integration/quibble] - 10https://gerrit.wikimedia.org/r/422193 (owner: 10Hashar) [17:44:40] (03Merged) 10jenkins-bot: Monkey patch $wgTmpDirectory [integration/quibble] - 10https://gerrit.wikimedia.org/r/422194 (owner: 10Hashar) [18:34:36] PROBLEM - App Server Main HTTP Response on deployment-mediawiki07 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 hphp_invoke - string 'Wikipedia' not found on 'http://en.wikipedia.beta.wmflabs.org:80/wiki/Main_Page?debug=true' - 287 bytes in 0.006 second response time [18:35:58] PROBLEM - Puppet errors on deployment-etcd-01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [18:54:29] PROBLEM - Puppet errors on deployment-secureredirexperiment is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [19:08:17] (03PS6) 10Hashar: Docker container based on Stretch [integration/quibble] - 10https://gerrit.wikimedia.org/r/421900 [19:08:19] (03PS1) 10Hashar: Unhardcode Zuul --git-cache dir [integration/quibble] - 10https://gerrit.wikimedia.org/r/422224 [19:12:37] Krinkle: hi! When you get some spare time I could use your blessing to have webdriver io run Chrome in headless mode ( https://gerrit.wikimedia.org/r/#/c/358019/ ) [19:23:56] hasharDockerized: Is there a page that explains how Quibble fits into the CI architecture? [19:24:00] https://www.mediawiki.org/wiki/Continuous_integration [19:24:07] Some overview subpage? [19:24:20] Krinkle: nop. It is merely a proof of concept I will present to releng team next week [19:24:25] hey, i'm here to switch the deployment server, heh [19:24:33] and from there I guess it will get doc and a .plan to switch the CI jobs to it [19:24:34] unless it's too late for the train [19:24:38] should be transient though [19:33:20] Krinkle: in short the idea is to have a single script (quibble, in python) that concatenante the logic from Jenkins jobs, the slave scripts and whatever crap we have [19:33:58] hashar: Hm.. how does it relate to JJB? [19:34:20] I mean, will it still be triggered from a Jenkins job? [19:37:47] Krinkle: yup certainly [19:38:11] Krinkle: something like: ZUUL_PROJECT=mediawiki/extensions/Foo ... quibble --packages-sources vendor --db mysql [19:38:47] and that runs phpunit / karma / webdriver.io in a single container that has the required services (mysql, webserver, chromedriver) spawned in the background (and in the sam econtainer) [19:38:50] later on we don't know [19:39:04] most probably we would use an Helm chart to have kubernetes spawn whatever containers we need [19:39:16] and run stuff in that set of containers (== a "pod" in k8s) [19:39:25] but we are not there yet :] [19:39:36] so the idea of quibble is to unify all the CI logic for mediawiki at a single place [19:39:43] and long term migrate that to helm/k8s [19:39:54] and maybe replace mediawiki/vagrant entirely ( maybe ) [19:41:44] Krinkle: and in short https://gerrit.wikimedia.org/r/#/c/358019/ saves me from having to spawn Xvfb :] [19:42:19] which is only a couple lines to add really, but still I think using chromium in headless mode is a good thing overall [19:43:28] for karma it is a bit more nasty: unset(DISPLAY); CHROMIUM_FLAGS='--headless --disable-gpu --remote-debugging-port=9222' [19:43:38] but probably karma has a ChromiumHeadless or something like that [20:00:31] (03CR) 10BearND: [C: 031] Whitelist Jberkel [integration/config] - 10https://gerrit.wikimedia.org/r/422147 (owner: 10Mholloway) [20:01:45] (03CR) 10Hashar: [C: 032] Whitelist Jberkel [integration/config] - 10https://gerrit.wikimedia.org/r/422147 (owner: 10Mholloway) [20:02:17] RECOVERY - Puppet staleness on deployment-maps01 is OK: OK: Less than 1.00% above the threshold [3600.0] [20:02:58] (03Merged) 10jenkins-bot: Whitelist Jberkel [integration/config] - 10https://gerrit.wikimedia.org/r/422147 (owner: 10Mholloway) [20:03:26] hashar: Ah, okay, for mw stuff, not all JJB in general. [20:04:39] Krinkle: yeah the scope is limited to mediawiki core and related (extensions, skins, vendor) [20:04:47] which is already a large scope hehe [20:06:37] Yippee, build fixed! [20:06:38] Project mwext-phpunit-coverage-publish build #2639: 09FIXED in 2 min 34 sec: https://integration.wikimedia.org/ci/job/mwext-phpunit-coverage-publish/2639/ [20:41:45] Krinkle: thanks for your feedback on https://gerrit.wikimedia.org/r/#/c/358019/5/tests/selenium/wdio.conf.js@87 . Will check with Zeljko tomorrow and get that merged :] [20:42:24] (03CR) 10Hashar: [C: 032] "Deployed!" [integration/config] - 10https://gerrit.wikimedia.org/r/422147 (owner: 10Mholloway) [20:46:51] 10Continuous-Integration-Config, 10Release-Engineering-Team (Kanban): Prepate CI for REL1_31 - https://phabricator.wikimedia.org/T190879#4086373 (10greg) p:05Triage>03Normal [21:06:04] 10Beta-Cluster-Infrastructure, 10User-zeljkofilipin, 10WorkType-NewFunctionality: Make selenium users use botflags at beta-cluster - https://phabricator.wikimedia.org/T116027#4086444 (10zhuyifei1999) >>! In T116027#3720104, @zeljkofilipin wrote: > @Kunal: I have just talked to @demon about this and he said y... [21:08:22] PROBLEM - Puppet errors on deployment-mx02 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [21:28:12] Does one have to be a member of the beta project to query it's db servers? Or is there some public interface for privacy-insensitive data (like mirrors in prod)? [21:29:56] there are no replicas of the beta dbs [21:30:41] Meh. Okay, thanks. [22:17:02] Does anyone know what piece of software is displaying pages like https://integration.wikimedia.org/ci/job/operations-puppet-catalog-compiler/10704/ [22:17:36] Whatever it is seems not to be in the 'jenkins' repo or in the 'jenkinsci' repo... [22:18:29] andrewbogott: by Jenkins repo do you mean upstream? [22:18:46] I mean https://gerrit.wikimedia.org/r/#/admin/projects/integration/jenkins [22:18:52] Ah ok [22:19:00] Yeh it’s puppet-compiler [22:19:25] andrewbogott: https://gerrit.wikimedia.org/r/#/admin/projects/operations/software/puppet-compiler [22:20:08] well, specifically, that page is displaying the string "No changes." and that string does not appear in that repo either [22:20:51] T190886 [22:20:51] T190886: operations-puppet-compiler UI says 'No changes' when there totally are changes - https://phabricator.wikimedia.org/T190886 [22:23:25] Ah [22:23:27] Yes [22:23:41] That was generated manually in Jenkins [22:24:05] (I’ve tried to open source it but it has not been merged yet) [22:24:42] andrewbogott: https://gerrit.wikimedia.org/r/#/c/325064/ [22:25:50] that patch is over my head but if it fixes that 'No changes' line then I'm all for it [22:26:23] Oh nope that doesn’t fix it [22:26:38] It just a copy of the Jenkins job and open sources it [22:26:51] So someone could make changes to it at a later point [22:27:47] 10Beta-Cluster-Infrastructure, 10Privacy: Flush private data on Beta Cluster - https://phabricator.wikimedia.org/T189541#4045055 (10demon) With AbuseFilter logging disabled in [[https://gerrit.wikimedia.org/r/#/c/421864/|gerrit 421864]] and CheckUser already disabled, I went ahead with truncating `cu_changes`... [22:46:14] The deployment server is now deploy1001.eqiad.wmnet and not tin anymore [22:46:21] deployment.eqiad.wmnet DNS name also changed now [22:48:13] 10Release-Engineering-Team (Watching / External), 10Operations, 10hardware-requests: eqiad: replacement tin/deployment server - https://phabricator.wikimedia.org/T174452#4086641 (10Dzahn) [22:48:19] 10Release-Engineering-Team (Watching / External), 10Operations, 10Patch-For-Review: setup/install/deploy deploy1001 as deployment server - https://phabricator.wikimedia.org/T175288#4086639 (10Dzahn) 05Open>03Resolved There has been a deploy from it, and all the changes above, incl. DNS service name, Muku... [22:49:02] 10Release-Engineering-Team (Watching / External), 10Operations, 10Patch-For-Review: setup/install/deploy deploy1001 as deployment server - https://phabricator.wikimedia.org/T175288#4086643 (10Dzahn) [22:53:19] 10Scap, 10Operations, 10Packaging, 10Patch-For-Review: Install git-lfs client (at least on scap targets & masters) - https://phabricator.wikimedia.org/T180628#4086673 (10Dzahn) [22:56:43] 10Scap, 10Operations, 10Packaging, 10Patch-For-Review: Install git-lfs client (at least on scap targets & masters) - https://phabricator.wikimedia.org/T180628#4086682 (10Dzahn) tin has been replaced by deploy1001 which is running stretch [deploy1001:~] $ apt-cache show git-lfs Package: git-lfs Version: 2.... [22:57:05] 10Beta-Cluster-Infrastructure, 10Privacy: Flush private data on Beta Cluster - https://phabricator.wikimedia.org/T189541#4086683 (10Tgr) >>! In T189541#4086602, @demon wrote: > I've killed the existing logfiles on flourine02, but have we ensured private data won't go here again? Do we need to disable something... [22:57:14] 10Release-Engineering-Team (Kanban): Develop canonical/single record of origin, machine readable list of all repos deployed to WMF sites. - https://phabricator.wikimedia.org/T190891#4086695 (10Jrbranaa) [22:58:20] 10Release-Engineering-Team (Kanban): Develop canonical/single record of origin, machine readable list of all repos deployed to WMF sites. - https://phabricator.wikimedia.org/T190891#4086708 (10Jrbranaa) @Aklapper, please add any use cases that might be of interest to you. I'll do the same. [23:01:08] 10Beta-Cluster-Infrastructure, 10Privacy: Flush private data on Beta Cluster - https://phabricator.wikimedia.org/T189541#4086709 (10demon) >>! In T189541#4086683, @Tgr wrote: >>>! In T189541#4086602, @demon wrote: >> I've killed the existing logfiles on flourine02, but have we ensured private data won't go her... [23:02:07] 10Release-Engineering-Team (Kanban): Develop canonical/single record of origin, machine readable list of all repos deployed to WMF sites. - https://phabricator.wikimedia.org/T190891#4086695 (10Dzahn) This is definitely not all, but one part of the puzzle is searching operations/puppet repo for "git::clone" to ge... [23:02:37] twentyafterfour, mutant: what repo does labs use for operations/debs/ wmf packages? [23:03:38] AaronSchulz: each operations/debs/foo is its own repo (not what you mean though?) [23:04:25] there isn't labs/debs/ it would use the same [23:04:49] afaict [23:05:39] git repo or apt repo? [23:05:42] * AaronSchulz is curious mutante: I don't mean the git one, but the built ones [23:05:44] 10Release-Engineering-Team (Kanban), 10Code-Health: Develop canonical/single record of origin, machine readable list of all repos deployed to WMF sites. - https://phabricator.wikimedia.org/T190891#4086714 (10Jrbranaa) [23:05:48] greg-g: apt [23:06:02] 10Release-Engineering-Team (Kanban), 10Wikimedia-Apache-configuration, 10Patch-For-Review: Cleanup remaining WikipediaMobileFirefoxOS references - https://phabricator.wikimedia.org/T187850#4086715 (10demon) >>! In T187850#4076945, @bd808 wrote: > The cleanup command @chasemp used was `rm -fR /usr/local/lib/... [23:06:24] afaik, the one prod uses apt.wm.o/stretch or whatever [23:09:35] AaronSchulz: ^ that, we just have apt.wikimedia.org https://apt.wikimedia.org/wikimedia/ [23:09:48] there used to be something about having local deb repos on cloud vps.. [23:10:37] using this https://wikitech.wikimedia.org/wiki/Aptly [23:10:55] that is if you wanted your own repo on a cloud VPS [23:12:51] greg-g: OK, that's what I expected [23:13:15] mutante: really I just want to get https://gerrit.wikimedia.org/r/#/c/407030/ (tiny change) onto the labs mc servers [23:13:32] the patches is merged in git already [23:14:25] somebody has to build it on boron.eqiad.wmnet. it's not fully automatic [23:15:31] and then it has to be copied from that to install1002 and imported with reprepro as a new version [23:19:29] i can look at that in general, but that would have to be tomorrow [23:20:40] mutante: that would be awesome [23:21:19] AaronSchulz: i added myself as reviewer to remind me [23:58:04] 10Release-Engineering-Team (Watching / External), 10DBA: Certain wiki databases missing from replicas? - https://phabricator.wikimedia.org/T132838#4086805 (10Krinkle)