[01:02:15] PROBLEM - Puppet staleness on deployment-restbase02 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [43200.0] [04:12:12] Yippee, build fixed! [04:12:13] Project mediawiki-core-code-coverage build #3070: 09FIXED in 1 hr 12 min: https://integration.wikimedia.org/ci/job/mediawiki-core-code-coverage/3070/ [05:21:38] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team, 10MediaWiki-Parser, 10Parsing-Team, 10Readers-Web-Backlog (Tracking): Templates rendering as links on beta cluster - https://phabricator.wikimedia.org/T173576#3686342 (10Tgr) Probably a difference in citation templates? Those do not get expanded... [09:01:34] (03CR) 10Hashar: [C: 032] Set file filter for apps-android-wikipedia-tox-jessie [integration/config] - 10https://gerrit.wikimedia.org/r/382837 (https://phabricator.wikimedia.org/T177016) (owner: 10Legoktm) [09:03:29] (03Merged) 10jenkins-bot: Set file filter for apps-android-wikipedia-tox-jessie [integration/config] - 10https://gerrit.wikimedia.org/r/382837 (https://phabricator.wikimedia.org/T177016) (owner: 10Legoktm) [09:03:50] (03CR) 10Hashar: "Deployed :)" [integration/config] - 10https://gerrit.wikimedia.org/r/382837 (https://phabricator.wikimedia.org/T177016) (owner: 10Legoktm) [09:05:06] (03CR) 10Hashar: [C: 032] Bump Android periodic test emulator to android-26 [integration/config] - 10https://gerrit.wikimedia.org/r/383619 (owner: 10Mholloway) [09:06:18] (03Merged) 10jenkins-bot: Bump Android periodic test emulator to android-26 [integration/config] - 10https://gerrit.wikimedia.org/r/383619 (owner: 10Mholloway) [09:12:07] PROBLEM - Free space - all mounts on integration-slave-jessie-android is CRITICAL: CRITICAL: integration.integration-slave-jessie-android.diskspace._mnt.byte_percentfree (No valid datapoints found)integration.integration-slave-jessie-android.diskspace.root.byte_percentfree (<44.44%) [09:18:14] (03CR) 10Hashar: "I have updated the job. The command being run is:" [integration/config] - 10https://gerrit.wikimedia.org/r/383619 (owner: 10Mholloway) [09:29:25] 10Release-Engineering-Team (Kanban), 10Phabricator, 10Regression: Remove useless/misleading "Comments" field on top of https://phabricator.wikimedia.org/maniphest/task/edit/form/1/ and other places - https://phabricator.wikimedia.org/T178068#3679470 (10jcrespo) Note this not only affects form/1/, but probabl... [09:43:47] (03PS2) 10Hashar: Provide Android SDK location as an argument to non-periodic test scripts [integration/config] - 10https://gerrit.wikimedia.org/r/368238 (https://phabricator.wikimedia.org/T171811) (owner: 10Mholloway) [09:44:37] (03CR) 10Hashar: [C: 032] "Deployed! :)" [integration/config] - 10https://gerrit.wikimedia.org/r/368238 (https://phabricator.wikimedia.org/T171811) (owner: 10Mholloway) [09:45:49] (03Merged) 10jenkins-bot: Provide Android SDK location as an argument to non-periodic test scripts [integration/config] - 10https://gerrit.wikimedia.org/r/368238 (https://phabricator.wikimedia.org/T171811) (owner: 10Mholloway) [09:46:18] 10Browser-Tests-Infrastructure, 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban), 10Wikidata, and 3 others: Run Wikibase daily browser tests on Jenkins - https://phabricator.wikimedia.org/T167432#3686766 (10zeljkofilipin) To make it explicit: - selenium-Wikibase-T167432 job runs... [09:54:47] (03CR) 10Hashar: "I ran" [integration/config] - 10https://gerrit.wikimedia.org/r/383619 (owner: 10Mholloway) [09:59:52] 10Browser-Tests-Infrastructure, 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban), 10Wikidata, and 3 others: Run Wikibase daily browser tests on Jenkins - https://phabricator.wikimedia.org/T167432#3686795 (10Tobi_WMDE_SW) ``` The save has failed. (failed-save) (MediawikiApi::ApiE... [10:04:49] 10Browser-Tests-Infrastructure, 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban), 10Wikidata, and 3 others: Run Wikibase daily browser tests on Jenkins - https://phabricator.wikimedia.org/T167432#3686814 (10Tobi_WMDE_SW) Another guess: We are not logging in to create new items (... [10:19:51] 10Browser-Tests-Infrastructure, 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban), 10Wikidata, and 3 others: Run Wikibase daily browser tests on Jenkins - https://phabricator.wikimedia.org/T167432#3686889 (10zeljkofilipin) I have tried debugging locally, but I get `Unrecognized v... [10:20:20] 10Gerrit, 10Readers-Web-Backlog, 10Patch-For-Review, 10Readers-Web-Kanban-Board, 10Unplanned-Sprint-Work: Temporarily allow pushing large objects - https://phabricator.wikimedia.org/T178189#3686890 (10ovasileva) p:05Triage>03High [10:33:50] PROBLEM - Mediawiki Error Rate on graphite-labs is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [10.0] [10:34:37] 10Browser-Tests-Infrastructure, 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban), 10Wikidata, and 3 others: Run Wikibase daily browser tests on Jenkins - https://phabricator.wikimedia.org/T167432#3686943 (10zeljkofilipin) >>! In T167432#3686814, @Tobi_WMDE_SW wrote: > We are not... [10:38:55] 10Browser-Tests-Infrastructure, 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban), 10Wikidata, and 3 others: Run Wikibase daily browser tests on Jenkins - https://phabricator.wikimedia.org/T167432#3686945 (10Tobi_WMDE_SW) For creating properties the user is logged in, see the cre... [10:48:50] PROBLEM - Mediawiki Error Rate on graphite-labs is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [10.0] [10:50:51] 10Browser-Tests-Infrastructure, 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban), 10Wikidata, and 3 others: Run Wikibase daily browser tests on Jenkins - https://phabricator.wikimedia.org/T167432#3686957 (10zeljkofilipin) I am able to reproduce the problem from my machine, targe... [11:07:24] 10Browser-Tests-Infrastructure, 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban), 10Wikidata, and 3 others: Run Wikibase daily browser tests on Jenkins - https://phabricator.wikimedia.org/T167432#3686975 (10Tobi_WMDE_SW) You could try to output the whole error object in the api-... [11:14:51] 10Browser-Tests-Infrastructure, 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban), 10Wikidata, and 3 others: Run Wikibase daily browser tests on Jenkins - https://phabricator.wikimedia.org/T167432#3686985 (10zeljkofilipin) {F10257452} [11:16:54] 10Browser-Tests-Infrastructure, 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban), 10Wikidata, and 3 others: Run Wikibase daily browser tests on Jenkins - https://phabricator.wikimedia.org/T167432#3687001 (10zeljkofilipin) The screenshot says: > Could not save due to an error. >... [11:22:01] 10Browser-Tests-Infrastructure, 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban), 10Wikidata, and 3 others: Run Wikibase daily browser tests on Jenkins - https://phabricator.wikimedia.org/T167432#3687003 (10zeljkofilipin) The error message is [[ https://phabricator.wikimedia.org... [11:47:33] 10Browser-Tests-Infrastructure, 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban), 10Wikidata, and 3 others: Run Wikibase daily browser tests on Jenkins - https://phabricator.wikimedia.org/T167432#3687032 (10Tobi_WMDE_SW) @aude @hoo @Addshore whoever has the rights to change this... [11:49:59] 10Continuous-Integration-Config, 10MediaWiki-General-or-Unknown: Make sure extensions using composer/npm for development dependencies have the right .gitignore rules - https://phabricator.wikimedia.org/T116434#3687036 (10hashar) p:05Triage>03Low [12:23:09] PROBLEM - Long lived cherry-picks on puppetmaster on deployment-puppetmaster02 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [12:44:47] 10Release-Engineering-Team (Kanban), 10Phabricator, 10Regression: Remove useless/misleading "Comments" field on top of https://phabricator.wikimedia.org/maniphest/task/edit/form/1/ and other places - https://phabricator.wikimedia.org/T178068#3687151 (10Huji) Correct. The "security issue" form (i.e. form 2) i... [13:04:55] Project selenium-Math » chrome,beta,Linux,BrowserTests build #546: 04FAILURE in 55 sec: https://integration.wikimedia.org/ci/job/selenium-Math/BROWSER=chrome,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=BrowserTests/546/ [13:04:57] Project selenium-Math » firefox,beta,Linux,BrowserTests build #546: 04FAILURE in 57 sec: https://integration.wikimedia.org/ci/job/selenium-Math/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=BrowserTests/546/ [13:32:56] 10Continuous-Integration-Config, 10Wikipedia-Android-App-Backlog, 10Patch-For-Review: Don't re-download the Android SDK for every (non-periodic) CI job - https://phabricator.wikimedia.org/T171811#3687277 (10Mholloway) 05Open>03Resolved [13:37:37] 10Browser-Tests-Infrastructure, 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban), 10Wikidata, and 3 others: Run Wikibase daily browser tests on Jenkins - https://phabricator.wikimedia.org/T167432#3687293 (10zeljkofilipin) I have submitter patch [[ https://gerrit.wikimedia.org/r/... [13:45:28] PROBLEM - App Server Main HTTP Response on deployment-mediawiki04 is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Service Unavailable - string 'Wikipedia' not found on 'http://en.wikipedia.beta.wmflabs.org:80/wiki/Main_Page?debug=true' - 1342 bytes in 0.003 second response time [13:45:36] PROBLEM - English Wikipedia Mobile Main page on beta-cluster is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Service Unavailable - string 'Wikipedia' not found on 'https://en.m.wikipedia.beta.wmflabs.org:443/wiki/Main_Page?debug=true' - 1993 bytes in 0.022 second response time [13:46:10] PROBLEM - English Wikipedia Main page on beta-cluster is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Service Unavailable - string 'Wikipedia' not found on 'https://en.wikipedia.beta.wmflabs.org:443/wiki/Main_Page?debug=true' - 1953 bytes in 0.020 second response time [13:50:37] RECOVERY - App Server Main HTTP Response on deployment-mediawiki04 is OK: HTTP OK: HTTP/1.1 200 OK - 46786 bytes in 5.676 second response time [13:50:37] 10Browser-Tests-Infrastructure, 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban), 10Wikidata, and 3 others: Run Wikibase daily browser tests on Jenkins - https://phabricator.wikimedia.org/T167432#3687360 (10Tobi_WMDE_SW) We should definitely use the user factory and somehow whit... [13:50:43] RECOVERY - English Wikipedia Mobile Main page on beta-cluster is OK: HTTP OK: HTTP/1.1 200 OK - 35299 bytes in 9.362 second response time [13:51:11] RECOVERY - English Wikipedia Main page on beta-cluster is OK: HTTP OK: HTTP/1.1 200 OK - 47364 bytes in 0.948 second response time [13:52:09] PROBLEM - App Server Main HTTP Response on deployment-mediawiki06 is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Service Unavailable - string 'Wikipedia' not found on 'http://en.wikipedia.beta.wmflabs.org:80/wiki/Main_Page?debug=true' - 1343 bytes in 0.005 second response time [13:52:45] 10Browser-Tests-Infrastructure, 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban), 10Wikidata, and 3 others: Run Wikibase daily browser tests on Jenkins - https://phabricator.wikimedia.org/T167432#3687363 (10zeljkofilipin) I have updated permissions for [[ https://en.wikipedia.be... [13:53:39] hashar: if you have some time, could you have a look at https://gerrit.wikimedia.org/r/#/c/383830/ (take your time!) [13:54:03] hashar: I think it is ready to be merged, but I've been known to be wrong when it is about JJB :) [13:55:24] gehel: doing the SWAT :\ [13:55:42] yeah, no emergency at all... [13:56:04] 10Browser-Tests-Infrastructure, 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban), 10Wikidata, and 3 others: Run Wikibase daily browser tests on Jenkins - https://phabricator.wikimedia.org/T167432#3687367 (10zeljkofilipin) Hm, seems it works if I use Selenium_user as both users.... [13:57:16] RECOVERY - App Server Main HTTP Response on deployment-mediawiki06 is OK: HTTP OK: HTTP/1.1 200 OK - 46796 bytes in 7.232 second response time [14:03:35] PROBLEM - Puppet errors on integration-r-lang-01 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [14:07:52] no_justification we will probaly want to set https://gerrit-review.googlesource.com/#/c/gerrit/+/133235/5/gerrit-sshd/src/main/java/com/google/gerrit/sshd/SshDaemon.java when ever we upgrade :). (Due to the amount of refs that are going up) [14:29:24] 10Release-Engineering-Team (Kanban), 10Phabricator, 10Regression: Remove useless/misleading "Comments" field on top of https://phabricator.wikimedia.org/maniphest/task/edit/form/1/ and other places - https://phabricator.wikimedia.org/T178068#3687520 (10mmodell) Should be fixed in {rPHABd606822} which I will... [14:29:40] 10Release-Engineering-Team (Kanban), 10Phabricator, 10Regression: Remove useless/misleading "Comments" field on top of https://phabricator.wikimedia.org/maniphest/task/edit/form/1/ and other places - https://phabricator.wikimedia.org/T178068#3687522 (10mmodell) [14:29:56] 10Release-Engineering-Team (Kanban), 10Phabricator, 10Regression: Phabricator admins can not edit all project fields - https://phabricator.wikimedia.org/T178107#3687523 (10mmodell) [14:30:08] 10Release-Engineering-Team (Kanban), 10Phabricator, 10Regression: Phabricator admins can not edit all project fields - https://phabricator.wikimedia.org/T178107#3680934 (10mmodell) hotfix incoming [14:35:10] 10Release-Engineering-Team (Kanban), 10Phabricator, 10Regression: Remove useless/misleading "Comments" field on top of https://phabricator.wikimedia.org/maniphest/task/edit/form/1/ and other places - https://phabricator.wikimedia.org/T178068#3687548 (10mmodell) 05Open>03Resolved [14:36:13] 10Release-Engineering-Team (Kanban), 10Phabricator, 10Regression: Phabricator admins can not edit all project fields - https://phabricator.wikimedia.org/T178107#3687554 (10mmodell) 05Open>03Resolved [14:36:39] 10Continuous-Integration-Config, 10Wikipedia-Android-App-Backlog, 10Patch-For-Review: Limit Android CI jobs to running only when relevant files change - https://phabricator.wikimedia.org/T177016#3687557 (10Mholloway) 05Open>03Resolved a:03Mholloway [14:43:35] RECOVERY - Puppet errors on integration-r-lang-01 is OK: OK: Less than 1.00% above the threshold [0.0] [14:45:31] 10Continuous-Integration-Infrastructure (shipyard), 10Operations, 10Patch-For-Review, 10User-Joe: Unify production and CI docker image build process - https://phabricator.wikimedia.org/T177276#3687595 (10Addshore) @thcipriani having run with dates for the past weeks I really don't like them. It would make... [14:55:22] my patch in config repo that got merged around one hour ago has not landed in beta, I checked deployment-tin and it's not there. Can I do it manually or also is there a place to check what's the status of these deploys? [14:57:25] (03PS1) 10Mholloway: Run apps-android-wikipedia-npm-node-6-jessie only when js files change [integration/config] - 10https://gerrit.wikimedia.org/r/384531 (https://phabricator.wikimedia.org/T177016) [15:01:22] it's probably this: https://integration.wikimedia.org/ci/view/Beta/ [15:19:02] Amir1: ah, yeah, sometimes beta gets a little stuck, https://www.mediawiki.org/wiki/Continuous_integration/Jenkins#Hung_beta_code.2Fdb_update I'll try to unstick it and get updates out the door. [15:19:36] thank you [15:19:43] let me know when you're done please [15:19:59] will do [15:23:50] Amir1: disconnect/reconnect dance preformed. automatic updates should start up again. https://integration.wikimedia.org/ci/job/beta-scap-eqiad/177783/console just started [15:45:50] RECOVERY - Mediawiki Error Rate on graphite-labs is OK: OK: Less than 1.00% above the threshold [1.0] [15:56:23] PROBLEM - puppet last run on contint1001 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [16:06:23] RECOVERY - puppet last run on contint1001 is OK: OK: Puppet is currently enabled, last run 1 second ago with 0 failures [16:08:14] (03PS12) 10Addshore: mwext-php70-phan-docker job [integration/config] - 10https://gerrit.wikimedia.org/r/381271 [16:12:31] (03PS1) 10Addshore: rename mwext-php70-phan-jessie-docker > mwext-php70-phan-docker [integration/config] - 10https://gerrit.wikimedia.org/r/384548 [16:12:40] (03CR) 10Addshore: [C: 032] rename mwext-php70-phan-jessie-docker > mwext-php70-phan-docker [integration/config] - 10https://gerrit.wikimedia.org/r/384548 (owner: 10Addshore) [16:14:16] (03CR) 10jerkins-bot: [V: 04-1] rename mwext-php70-phan-jessie-docker > mwext-php70-phan-docker [integration/config] - 10https://gerrit.wikimedia.org/r/384548 (owner: 10Addshore) [16:14:31] (03CR) 10jerkins-bot: [V: 04-1] rename mwext-php70-phan-jessie-docker > mwext-php70-phan-docker [integration/config] - 10https://gerrit.wikimedia.org/r/384548 (owner: 10Addshore) [16:15:31] (03PS2) 10Addshore: rename mwext-php70-phan-jessie-docker > mwext-php70-phan-docker [integration/config] - 10https://gerrit.wikimedia.org/r/384548 [16:15:44] (03CR) 10Addshore: [C: 032] rename mwext-php70-phan-jessie-docker > mwext-php70-phan-docker [integration/config] - 10https://gerrit.wikimedia.org/r/384548 (owner: 10Addshore) [16:16:55] (03Merged) 10jenkins-bot: rename mwext-php70-phan-jessie-docker > mwext-php70-phan-docker [integration/config] - 10https://gerrit.wikimedia.org/r/384548 (owner: 10Addshore) [16:25:35] (03PS1) 10Hashar: build.py: support dry run [integration/config] - 10https://gerrit.wikimedia.org/r/384553 [16:25:37] (03PS1) 10Hashar: build.py decouple methods [integration/config] - 10https://gerrit.wikimedia.org/r/384554 [16:35:28] (03CR) 10jerkins-bot: [V: 04-1] build.py: support dry run [integration/config] - 10https://gerrit.wikimedia.org/r/384553 (owner: 10Hashar) [16:36:28] (03CR) 10jerkins-bot: [V: 04-1] build.py decouple methods [integration/config] - 10https://gerrit.wikimedia.org/r/384554 (owner: 10Hashar) [16:41:18] hasharAway: I have a question :D about EXT_DEPENDENCIES [16:41:39] paladox: By default, 30s. [16:41:45] ^ I think that's a sane default :) [16:41:59] yep, though large installs had problems with that :) [16:42:18] Well, we'll start with the default then tweak as necessary [16:42:27] No need to preemptively change it without data :) [16:42:35] ok :). [16:44:17] infact, none of the zuul env vars seem to get set for https://integration.wikimedia.org/ci/job/mwext-php70-phan-docker/19/console [16:45:15] im a failure, ignore me [16:49:18] no_justification apparently the steps to disable a user in gerrit, get complicated now that you have to either use the ssh command or rest api. Or go git cloning the All-Users repo and search through all the refs for the mantiching id. :). [16:57:23] I use the ssh command anyway [16:57:26] So, that's easy [17:01:46] 10Gerrit, 10Readers-Web-Backlog, 10Patch-For-Review, 10Readers-Web-Kanban-Board, 10Unplanned-Sprint-Work: Temporarily allow pushing large objects - https://phabricator.wikimedia.org/T178189#3688117 (10Legoktm) >>! In T178189#3684901, @bmansurov wrote: >>>! In T178189#3684715, @Legoktm wrote: >> Just to c... [17:02:43] (03PS13) 10Addshore: mwext-php70-phan-docker job [integration/config] - 10https://gerrit.wikimedia.org/r/381271 [17:03:01] hasharAway: legoktm ^^ that works now, but it needs to git cache on the docker slaves for core [17:08:57] 10Release-Engineering-Team (Kanban), 10Release Pipeline (Blubber): Include Blubber metadata in Dockerfile output as labels - https://phabricator.wikimedia.org/T178022#3688153 (10dduvall) p:05Triage>03Normal [17:09:41] !log addshore@integration-slave-docker-c2-m4-d40-1005:/srv/git/mediawiki$ sudo git clone --bare https://gerrit.wikimedia.org/r/p/mediawiki/core.git [17:09:46] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [17:09:51] 10Release-Engineering-Team (Kanban), 10Release Pipeline, 10Patch-For-Review: Define new Jenkins pipeline for container build phase - https://phabricator.wikimedia.org/T175297#3688157 (10thcipriani) [17:09:53] 10Release-Engineering-Team (Kanban), 10Release Pipeline, 10Patch-For-Review: Establish secure way of passing registry credentials from Jenkins to Docker - https://phabricator.wikimedia.org/T176896#3688155 (10thcipriani) 05Open>03Resolved a:03dduvall [17:10:29] 10Release-Engineering-Team (Kanban), 10Release Pipeline, 10Patch-For-Review: Define new Jenkins pipeline for container build phase - https://phabricator.wikimedia.org/T175297#3589305 (10thcipriani) [17:10:31] 10Release-Engineering-Team (Kanban), 10Release Pipeline, 10Patch-For-Review: Find CI container build location - https://phabricator.wikimedia.org/T173128#3688160 (10thcipriani) 05Open>03Resolved [17:11:23] woo, 40 second phan run! https://integration.wikimedia.org/ci/job/mwext-php70-phan-docker/24/console vs 1 min 30 https://integration.wikimedia.org/ci/job/mwext-php70-phan-jessie/5012/consoleFull [17:11:45] 10Release-Engineering-Team (Kanban), 10Release Pipeline, 10Patch-For-Review: Define new Jenkins pipeline for container build phase - https://phabricator.wikimedia.org/T175297#3688162 (10dduvall) 05stalled>03Open [17:11:52] addshore: I am not around. But I got a puppet patch to add oeprations/puppet to /srv/git as a bare repo :) [17:12:04] hasharAway: I just saw it! :) [17:12:15] I just manually did core on the 1 slave that im trying phan jobs on for now [17:12:43] addshore: https://gerrit.wikimedia.org/r/#/c/383843/3 feel free to amend or add another change on top of it [17:12:49] addshore: you might want mediawiki/vendor as well [17:12:57] ack! [17:13:11] need to make EXT_DEPENDENCIES work with this job first, unsure why it isnt [17:13:11] and I have yet to find a way to safely update them from time to time :] [17:13:30] EXT_DEPENDENCIES is injected by Zuul as a a build parameter [17:13:43] it is then an env variable. Probably need to add it to the docker env file [17:13:49] zuul-env: or something like that [17:14:15] 10Release-Engineering-Team, 10Release Pipeline: Pipeline image build cleanup - https://phabricator.wikimedia.org/T177867#3688169 (10dduvall) [17:14:28] addshore: docker-zuul-env in jjb/macro-docker.yaml [17:14:33] dinner time & [17:14:38] 10Release-Engineering-Team (Next), 10Release Pipeline: Secret storage on contint1001 for Docker registry password - https://phabricator.wikimedia.org/T175298#3688174 (10thcipriani) 05Open>03Resolved a:03thcipriani [17:14:41] 10Release-Engineering-Team, 10Release Pipeline: Pipeline image build cleanup - https://phabricator.wikimedia.org/T177867#3673142 (10dduvall) p:05Triage>03Normal [17:14:43] 10Release-Engineering-Team (Kanban), 10Release Pipeline, 10Patch-For-Review: Establish secure way of passing registry credentials from Jenkins to Docker - https://phabricator.wikimedia.org/T176896#3688176 (10thcipriani) [17:15:05] ack [17:15:08] I'm doing: [17:15:08] - shell: "echo -e EXT_DEPENDENCIES=$EXT_DEPENDENCIES >> .env" [17:15:18] but $EXT_DEPENDENCIES always seems to be empty :P [17:25:15] aaaah [17:25:39] the prefix in that python is "mwext-php70-phan-jessie" but this job is called "mwext-php70-phan-docker", so then env vars are not added [17:26:22] (03PS14) 10Addshore: mwext-php70-phan-docker job [integration/config] - 10https://gerrit.wikimedia.org/r/381271 [17:26:56] 10Browser-Tests-Infrastructure, 10releng-201718-q1, 10MediaWiki-General-or-Unknown, 10Epic, and 6 others: Port Selenium tests from Ruby to Node.js - https://phabricator.wikimedia.org/T139740#3688198 (10greg) [17:27:25] 10Release-Engineering-Team (Kanban), 10Mathoid, 10Release Pipeline: Add experimental blubber test build/run to mathoid jenkins test pipeline - https://phabricator.wikimedia.org/T177954#3688200 (10thcipriani) p:05Triage>03Normal a:03thcipriani [17:27:39] (03PS1) 10Addshore: zuul parameter_functions mwext-php70-phan-jessie* to mwext-php70-phan* [integration/config] - 10https://gerrit.wikimedia.org/r/384565 [17:27:52] (03CR) 10Addshore: [C: 032] zuul parameter_functions mwext-php70-phan-jessie* to mwext-php70-phan* [integration/config] - 10https://gerrit.wikimedia.org/r/384565 (owner: 10Addshore) [17:29:06] (03Merged) 10jenkins-bot: zuul parameter_functions mwext-php70-phan-jessie* to mwext-php70-phan* [integration/config] - 10https://gerrit.wikimedia.org/r/384565 (owner: 10Addshore) [17:29:45] !log reloaded zuul for https://gerrit.wikimedia.org/r/384565 [17:29:50] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [17:30:14] 10Release-Engineering-Team (Next), 10Patch-For-Review, 10Release Pipeline (Blubber): Experiment with blubber/helm config in mathoid - https://phabricator.wikimedia.org/T173127#3688208 (10dduvall) [17:30:35] mhhm, maybe a reload is not enough [17:30:51] 10Release-Engineering-Team (Next), 10Release Pipeline (Blubber): Blubber config input validation - https://phabricator.wikimedia.org/T175186#3688211 (10thcipriani) [17:31:24] * addshore will wait for hasharAway :D [18:11:56] (03PS2) 10Hashar: build.py decouple find_tree from gen_deps_tree [integration/config] - 10https://gerrit.wikimedia.org/r/384554 [18:11:58] (03PS1) 10Hashar: build.py: split gen_deps_tree in two functions [integration/config] - 10https://gerrit.wikimedia.org/r/384573 [18:17:53] hasharAway, back? ;) [18:23:14] (03CR) 10jerkins-bot: [V: 04-1] build.py decouple find_tree from gen_deps_tree [integration/config] - 10https://gerrit.wikimedia.org/r/384554 (owner: 10Hashar) [18:23:16] (03CR) 10jerkins-bot: [V: 04-1] build.py: split gen_deps_tree in two functions [integration/config] - 10https://gerrit.wikimedia.org/r/384573 (owner: 10Hashar) [18:26:04] PROBLEM - Free space - all mounts on integration-slave-jessie-1002 is CRITICAL: CRITICAL: integration.integration-slave-jessie-1002.diskspace._mnt.byte_percentfree (No valid datapoints found)integration.integration-slave-jessie-1002.diskspace._srv.byte_percentfree (<100.00%) [18:36:16] 10Gerrit, 10Operations: Enable auto submodule updates on operations/puppet - https://phabricator.wikimedia.org/T178322#3688396 (10Paladox) [18:38:33] 10Gerrit, 10Operations, 10Patch-For-Review: Enable auto submodule updates on operations/puppet - https://phabricator.wikimedia.org/T178322#3688429 (10demon) 05Open>03declined I'm pretty sure nobody in ops wants this functionality on puppet -- the parent repo operates by the policy of "if you merge it, yo... [19:04:53] addshore: I am sprinting some tweaks to build.py [19:05:20] How do i get zuul to use my updated parameter_functions.py? [19:05:26] does it need a 'full' restart? [19:06:43] reload should have worked :). [19:07:18] mhhhm, ill have another look after dinner then! [19:08:51] (03PS2) 10Hashar: build.py: support dry run [integration/config] - 10https://gerrit.wikimedia.org/r/384553 [19:08:53] (03PS3) 10Hashar: build.py decouple find_tree from gen_deps_tree [integration/config] - 10https://gerrit.wikimedia.org/r/384554 [19:08:55] (03PS2) 10Hashar: build.py: split gen_deps_tree in two functions [integration/config] - 10https://gerrit.wikimedia.org/r/384573 [19:08:59] (03PS1) 10Hashar: build.py: refactor dep tree to use 'scratch' [integration/config] - 10https://gerrit.wikimedia.org/r/384580 [19:09:10] addshore: fab deploy_zuul ? [19:09:14] addshore: it is usually enough [19:09:34] hmm, okay, then i must be having a different issue! [19:10:09] addshore: then maybe I am wrong and it really needs a restart :D [19:10:30] addshore: ohh [19:10:40] addshore: and to have the parameters injected, the job has to be triggerd by zuul [19:10:47] so you cant manually run it from the jenkins ui [19:11:25] hmm, so a rerun of a previous job wouldnt do it? [19:11:30] I'lll try that! [19:12:06] PROBLEM - Puppet errors on deployment-mathoid is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [19:17:31] hasharAway: yup, that was it! [19:18:09] addshore: yeah the parameter changed and the previous job saved in jenkins had an obsolete version [19:21:27] (03CR) 10Addshore: [V: 031] "https://integration.wikimedia.org/ci/job/mwext-php70-phan-docker/31/console" [integration/config] - 10https://gerrit.wikimedia.org/r/381271 (owner: 10Addshore) [19:21:58] sweet, well, I have finally finished phan for extensions then! doing the ore job should be pretty easy too! [19:23:35] thcipriani: fyi a bit of refactor that effects nodepool I expect nothing to happen https://gerrit.wikimedia.org/r/#/c/384582/ (just a heads up because...life) [19:24:25] chasemp: werd. Good looking-out, thanks for the heads-up :) [19:40:56] (03CR) 10Addshore: [V: 031 C: 032] mwext-php70-phan-docker job [integration/config] - 10https://gerrit.wikimedia.org/r/381271 (owner: 10Addshore) [19:41:17] 10Gerrit, 10Upstream: Gerrit should feature customizable message on Login page (No 'Forgot password' link in the gerrit login page.) - https://phabricator.wikimedia.org/T60205#3688678 (10Paladox) I would like to get upstream to support logging in / registering through rest api. That will allow us to get polyge... [19:42:59] (03Merged) 10jenkins-bot: mwext-php70-phan-docker job [integration/config] - 10https://gerrit.wikimedia.org/r/381271 (owner: 10Addshore) [19:45:22] Bah, one more isssue it would seem! [19:48:36] no_justification sorry for creating https://gerrit.wikimedia.org/r/384588 but i would have forgot otherwise. I've been using mariadb locally. I had to manually modify puppet code to do it. So i created a patch. :) [19:48:59] I guess we should finish the systemd crap [19:49:18] And swap for the scap-deployed version [19:49:40] I've already finished it no_justification :) [19:49:46] it's waiting for someone to merge [19:49:54] i've been using it for the last couple of weeks [19:50:01] systemd puppet change [19:50:17] https://gerrit.wikimedia.org/r/#/c/378768/ [19:50:36] That's not finished :) [19:50:40] Until it's merged [19:50:44] o/ releng folks [19:50:55] I'm trying to figure out what awight was doing in our beta deploy. [19:51:08] When I go look at sca03, I see this in our gitmodules: https://pastebin.ca/3888405 [19:51:12] oh heh :) [19:51:19] Does that diff mean anything to anyone? [19:52:05] RECOVERY - Puppet errors on deployment-mathoid is OK: OK: Less than 1.00% above the threshold [0.0] [19:52:31] halfak: Yes. We rewrite your gitmodules for target machines -- they don't all fetch from Gerrit/Phab/whatever directly [19:52:45] OK so it's safe to ignore? [19:53:13] Yes [19:53:16] Unless it's broken :) [19:53:18] cool thanks :) [19:53:42] yw [19:57:00] !log deploying ores 0f3fe9f T175180 [19:57:05] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [19:57:05] T175180: Deploy ORES (revscoring 2.0) - https://phabricator.wikimedia.org/T175180 [20:02:14] no_justification, https://phabricator.wikimedia.org/P6132 [20:02:17] Looks like it did fail. [20:02:51] PROBLEM - Free space - all mounts on deployment-sca03 is CRITICAL: CRITICAL: deployment-prep.deployment-sca03.diskspace._srv.byte_percentfree (<40.00%) [20:02:57] Looks like it's only the one submodule [20:03:03] I wonder if disk space is an issue ^? [20:03:08] Not clear to me what happened. [20:03:16] Failed writing body (974 != 5792) [20:03:20] That seems suspicious [20:04:02] 3G available on /srv of sca03 [20:05:48] demon@deployment-sca03:/srv/deployment/ores/deploy-cache/revs$ du -sh * [20:05:48] 2.7G 0f3fe9f1919ffefd2c571b5fdcfe358f93d5c5c4 [20:05:48] 3.0G 1d35aa5b853f304bb11dd46bc79dfc3660f68ce8 [20:05:48] 3.0G 42c56632e36827f0fe898d010fd8664f800d8900 [20:05:48] 3.0G 835d848131c1f9c9628aa85e36630b8f2f02df67 [20:05:48] 3.5G e17f7c4db5aeb44eca37fe341a70898fe2a134eb [20:05:53] Well each deploy is ~3gb [20:05:58] So, we might need to clean these up [20:06:28] hppy for that. [20:06:31] Delete away! [20:06:35] cf T137124 [20:06:36] T137124: Scap3 submodule space issues - https://phabricator.wikimedia.org/T137124 [20:08:06] * no_justification tries some aggressive repacks first [20:09:39] (03PS1) 10Addshore: final umask fixes for mwext-php70-phan-docker job [integration/config] - 10https://gerrit.wikimedia.org/r/384597 [20:09:47] /dev/mapper/vd-second--local--disk 21G 11G 8.8G 54% /srv [20:10:20] !log deployment-prep Dropped 2 deploy-cache entries for ORES from deployment-sca03 [20:10:25] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [20:10:34] Thanks no_justification [20:10:35] !log deployment-prep Both repos date from July [20:10:36] trying again [20:10:39] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [20:11:14] !log deploying ores 0f3fe9f T175180 (second attempt) [20:11:18] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [20:11:19] T175180: Deploy ORES (revscoring 2.0) - https://phabricator.wikimedia.org/T175180 [20:12:50] RECOVERY - Free space - all mounts on deployment-sca03 is OK: OK: All targets OK [20:16:21] thcipriani: thoughts on having umask 0002 for all the docker based stuff? right now in the phan image I just do RUN echo "umask 0002" >> /etc/bash.bashrc [20:17:33] 10Gerrit, 10Release-Engineering-Team (Kanban), 10Scap (Tech Debt Sprint FY201718-Q2), 10ORES, and 3 others: Support git-lfs files in gerrit - https://phabricator.wikimedia.org/T171758#3688811 (10Paladox) I've done the configuration. We need to install the plugin. But the good thing is we can install things... [20:24:39] addshore: does that work? I wonder if we could set it in /etc/profile? [20:25:42] *has a go* [20:29:08] thcipriani: nop, https://integration.wikimedia.org/ci/job/mwext-php70-phan-docker/43/console [20:29:10] hmph [20:29:41] for the setup image I just set it in the entry point https://github.com/wikimedia/integration-config/blob/master/dockerfiles/ci-src-setup/setup-mw.sh#L5 maybe I should just do that agian [20:30:57] that's probably the most sure-fire way to do it [20:36:43] !log deploying ores 42c5663 T175180 (rolling back) [20:36:47] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [20:36:48] T175180: Deploy ORES (revscoring 2.0) - https://phabricator.wikimedia.org/T175180 [20:39:43] (03PS2) 10Addshore: final umask fixes for mwext-php70-phan-docker job [integration/config] - 10https://gerrit.wikimedia.org/r/384597 [20:40:21] (03CR) 10Addshore: [V: 031] "https://integration.wikimedia.org/ci/job/mwext-php70-phan-docker/47/console" [integration/config] - 10https://gerrit.wikimedia.org/r/384597 (owner: 10Addshore) [20:45:49] (03PS3) 10Hashar: build.py: support dry run [integration/config] - 10https://gerrit.wikimedia.org/r/384553 [20:45:51] (03PS4) 10Hashar: build.py decouple find_tree from gen_deps_tree [integration/config] - 10https://gerrit.wikimedia.org/r/384554 [20:45:53] (03PS3) 10Hashar: build.py: split gen_deps_tree in two functions [integration/config] - 10https://gerrit.wikimedia.org/r/384573 [20:45:55] (03PS2) 10Hashar: build.py: refactor dep tree to use 'scratch' [integration/config] - 10https://gerrit.wikimedia.org/r/384580 [20:48:55] !log deploying ores fb55ab8 T175180 (fixes eswiki) [20:48:59] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [20:49:00] T175180: Deploy ORES (revscoring 2.0) - https://phabricator.wikimedia.org/T175180 [20:53:08] no_justification, hit the disk space limit again :( [20:53:44] (did a rollback and a new deploy) [20:54:58] You still have 7.9G [20:55:42] oh... hmm [20:55:47] * halfak looks more [20:55:49] sorry [20:59:21] "fatal: reference is not a tree: fb55ab80c6f544e57dd1d56e0713047d046c5275" [20:59:28] WTF [20:59:32] do git submodule too [20:59:39] Did that [20:59:53] All works as expected on deployment-tin [21:00:00] but then fails on sca03 [21:01:58] (03PS3) 10Addshore: final umask & ENV fixes for mwext-php70-phan-docker job [integration/config] - 10https://gerrit.wikimedia.org/r/384597 [21:02:11] thcipriani: awesome, think that is all done :) [21:02:22] nice :) [21:02:41] the last thing that was missing was a bunch more zuul env vars, should probably reword docker-zuul-env a bit [21:02:49] I'm trying to do it and got the same situation [21:04:14] Amir1: halfak somehow the git checkout in deployment-sca03:/srv/deployment/ores/deploy-cache/revs/fb55ab80c6f544e57dd1d56e0713047d046c5275 doesn't contain fb55ab80c6f544e57dd1d56e0713047d046c5275 [21:04:41] thcipriani, maybe something was delayed? I tried to grab it right after submitting the patchset [21:05:12] hrm maybe? try it now with that commit [21:05:34] * halfak tries again [21:09:16] (03PS4) 10Addshore: final umask & ENV fixes for mwext-php70-phan-docker job [integration/config] - 10https://gerrit.wikimedia.org/r/384597 [21:09:17] looks like we deployed! [21:09:21] thanks thcipriani [21:09:26] * thcipriani doffs hat [21:09:36] Amir1, we're online now [21:09:43] What do the logs look like in MediaWiki [21:09:45] * halfak edits [21:09:52] awesome [21:11:46] brb [21:17:37] (03PS5) 10Addshore: final umask & ENV fixes for mwext-php70-phan-docker job [integration/config] - 10https://gerrit.wikimedia.org/r/384597 [21:20:17] (03Abandoned) 10Addshore: docker: stop passing HOME [integration/config] - 10https://gerrit.wikimedia.org/r/380962 (owner: 10Hashar) [21:20:40] back now [21:20:53] halfak: it seems up [21:23:59] (03PS1) 10Addshore: Switch from mwext-php70-phan-jessie to mwext-php70-phan-docker [integration/config] - 10https://gerrit.wikimedia.org/r/384614 [21:25:08] (03PS1) 10Addshore: Remove mwext-php70-phan-jessie [integration/config] - 10https://gerrit.wikimedia.org/r/384615 [21:32:07] (03CR) 10Jdlrobson: Run Selenium tests for Reading Web extension (033 comments) [integration/config] - 10https://gerrit.wikimedia.org/r/384041 (https://phabricator.wikimedia.org/T162256) (owner: 10Zfilipin) [21:36:40] (03PS2) 10Chad: make-wmf-branch: Stop amending commits for branches with submodules [tools/release] - 10https://gerrit.wikimedia.org/r/380885 (https://phabricator.wikimedia.org/T175324) [21:52:44] https://phab-01.wmflabs.org/ now is now https://phab.wmflabs.org/ [21:53:57] twentyafterfour ^^ [22:03:23] no_justification hi, wondering could you put a +1 or -1 on https://gerrit.wikimedia.org/r/#/c/378768/ please. Just needs your review to be considered to be merged :). [22:12:17] thanks :). Comments addressed now :). [22:13:41] (03CR) 10Thcipriani: make-wmf-branch: Stop amending commits for branches with submodules (031 comment) [tools/release] - 10https://gerrit.wikimedia.org/r/380885 (https://phabricator.wikimedia.org/T175324) (owner: 10Chad) [22:28:39] 10Gerrit: Add an icon to a patchset in a changeset view when the patchset has attached comments - https://phabricator.wikimedia.org/T52600#3689146 (10Paladox) This is possible now with polygerrit using a polymer element for example https://gerrit-review.googlesource.com/#/c/gerrit/+/133551/3/polygerrit-ui/app/el... [22:49:54] (03PS3) 10Chad: make-wmf-branch: Stop amending commits for branches with submodules [tools/release] - 10https://gerrit.wikimedia.org/r/380885 (https://phabricator.wikimedia.org/T175324) [22:52:57] i've noticed that making changes in operations/puppet is really slow [22:53:01] takes a few secs [22:53:12] 5-10 secs [22:54:13] paladox: you mean ssh:// not https:// ? [22:54:20] ssh [22:54:44] but it happends through inline editing [22:54:46] or rebasing [22:54:51] hm [22:55:05] it must have alot of refs [23:01:50] (03CR) 10Thcipriani: [C: 032] make-wmf-branch: Stop amending commits for branches with submodules [tools/release] - 10https://gerrit.wikimedia.org/r/380885 (https://phabricator.wikimedia.org/T175324) (owner: 10Chad) [23:03:56] (03Merged) 10jenkins-bot: make-wmf-branch: Stop amending commits for branches with submodules [tools/release] - 10https://gerrit.wikimedia.org/r/380885 (https://phabricator.wikimedia.org/T175324) (owner: 10Chad) [23:40:04] 10Release-Engineering-Team (Kanban), 10MediaWiki-Release-Tools, 10Patch-For-Review: It shouldn't be possible to create WMF branches on master - https://phabricator.wikimedia.org/T175324#3689302 (10demon) 05Open>03Resolved Should be fixed now.