[02:16:54] (03CR) 10Chad: [C: 04-2] "I'm vetoing this. In an ideal world we don't make /any/ changes to repos when we branch. This moves backwards from that." [tools/release] - 10https://gerrit.wikimedia.org/r/376658 (https://phabricator.wikimedia.org/T175324) (owner: 10MaxSem) [02:53:43] Project beta-scap-eqiad build #172787: 04FAILURE in 0.36 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/172787/ [03:06:25] Yippee, build fixed! [03:06:26] Project beta-scap-eqiad build #172788: 09FIXED in 2 min 44 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/172788/ [03:22:56] PROBLEM - Puppet errors on deployment-mira is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [04:02:53] RECOVERY - Puppet errors on deployment-mira is OK: OK: Less than 1.00% above the threshold [0.0] [04:06:28] Yippee, build fixed! [04:06:29] Project mediawiki-core-code-coverage build #3004: 09FIXED in 1 hr 6 min: https://integration.wikimedia.org/ci/job/mediawiki-core-code-coverage/3004/ [04:41:42] (03Abandoned) 10MaxSem: make-wmf-branch: update .gitreview [tools/release] - 10https://gerrit.wikimedia.org/r/376658 (https://phabricator.wikimedia.org/T175324) (owner: 10MaxSem) [04:53:50] PROBLEM - Mediawiki Error Rate on graphite-labs is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [10.0] [05:38:48] PROBLEM - Mediawiki Error Rate on graphite-labs is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [10.0] [06:56:38] 10Continuous-Integration-Config, 10MinusX: Add MinusX to MediaWiki extensions and other repositories - https://phabricator.wikimedia.org/T175794#3603339 (10Legoktm) [06:57:19] PROBLEM - Puppet errors on deployment-kafka01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [07:14:00] 10Continuous-Integration-Config, 10MinusX, 10Patch-For-Review: Reject non-executable files with execute bits with a build check - https://phabricator.wikimedia.org/T168659#3603360 (10Legoktm) 05Open>03Resolved I've announced the creationg of #minusx to wikitech-l: https://lists.wikimedia.org/pipermail/wi... [07:31:45] (03PS1) 10Hashar: Migrate php-compile-php55 to Jessie [integration/config] - 10https://gerrit.wikimedia.org/r/377712 (https://phabricator.wikimedia.org/T161882) [07:34:33] RECOVERY - Puppet staleness on deployment-kafka01 is OK: OK: Less than 1.00% above the threshold [3600.0] [07:38:36] (03CR) 10Hashar: "works!" [integration/config] - 10https://gerrit.wikimedia.org/r/377712 (https://phabricator.wikimedia.org/T161882) (owner: 10Hashar) [07:40:19] !log jenkins: on nodes, removing the labels phpflavor-* they are no more needed - T 161882 [07:40:23] !log jenkins: on nodes, removing the labels phpflavor-* they are no more needed - T161882 [07:40:23] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [07:40:26] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [07:40:27] T161882: Migrate PHP5.5 jobs from Trusty to Jessie - https://phabricator.wikimedia.org/T161882 [07:43:48] (03PS1) 10Addshore: docker: use wmfreleng/mediawiki-extensions-phan:v2017.09.11.19.08 [integration/config] - 10https://gerrit.wikimedia.org/r/377714 [07:44:42] 10Continuous-Integration-Infrastructure (phase-out-trusty), 10Release-Engineering-Team (Kanban), 10Patch-For-Review: Remove trusty from Nodepool and clean out puppet - https://phabricator.wikimedia.org/T175696#3603419 (10hashar) [07:44:45] 10Continuous-Integration-Infrastructure (phase-out-trusty), 10Release-Engineering-Team (Kanban), 10Patch-For-Review: Migrate PHP5.5 jobs from Trusty to Jessie - https://phabricator.wikimedia.org/T161882#3603418 (10hashar) 05Open>03Resolved [07:45:12] (03PS1) 10Hashar: zuul: remove references to trusty [integration/config] - 10https://gerrit.wikimedia.org/r/377715 [07:46:56] (03CR) 10Addshore: [C: 032] "deployed" [integration/config] - 10https://gerrit.wikimedia.org/r/377714 (owner: 10Addshore) [07:47:08] hashar: woohoo, congrats on getting rid of trusty! :) [07:47:35] legoktm: thank you :)]]] [07:47:51] legoktm: all made possible as soon as I managed to compile php5.5 for jessie [07:48:02] (03Merged) 10jenkins-bot: docker: use wmfreleng/mediawiki-extensions-phan:v2017.09.11.19.08 [integration/config] - 10https://gerrit.wikimedia.org/r/377714 (owner: 10Addshore) [07:48:04] it took me a while to figure how sury.org was compiling them, but eventually I found the source :D [07:48:09] next step: switch everything to docker [07:48:48] 10Beta-Cluster-Infrastructure: Deployment wiki is flooded by spam and should be cleaned up, perhaps even restricted more - https://phabricator.wikimedia.org/T175197#3603422 (10Sau226) I've given you global sysop so you can deal with possible spam network wide @Mainframe98 . You will now be able to set blocks on... [07:48:56] (03PS8) 10Addshore: docker: mediawiki-extensions-phan image [integration/config] - 10https://gerrit.wikimedia.org/r/371708 [07:49:03] (03PS1) 10Hashar: maven jobs: update JDK name [integration/config] - 10https://gerrit.wikimedia.org/r/377716 [07:49:13] (03CR) 10jerkins-bot: [V: 04-1] docker: mediawiki-extensions-phan image [integration/config] - 10https://gerrit.wikimedia.org/r/371708 (owner: 10Addshore) [07:49:46] !log Jenkins: removing the Ubuntu JDK from https://integration.wikimedia.org/ci/configureTools/ [07:49:50] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [07:50:35] 10Continuous-Integration-Config, 10Release-Engineering-Team (Backlog): Switch MediaWiki coverage job from PHP 5 to PHP 7 - https://phabricator.wikimedia.org/T147778#3603423 (10hashar) It is not running on trusty. [07:51:15] (03PS3) 10Addshore: docker: use tox --notest when populating cache [integration/config] - 10https://gerrit.wikimedia.org/r/369605 (owner: 10Hashar) [07:51:18] (03PS8) 10Addshore: docker: puppet, shallow fetches & no second clone [integration/config] - 10https://gerrit.wikimedia.org/r/374507 [07:51:31] (03PS6) 10Addshore: docker: Tag images when building with a date stamp [integration/config] - 10https://gerrit.wikimedia.org/r/377249 [07:51:39] 10Continuous-Integration-Infrastructure (phase-out-trusty), 10Release-Engineering-Team (Backlog), 10Patch-For-Review: mysql does not start when Trusty instances spawn - https://phabricator.wikimedia.org/T141450#3603425 (10hashar) 05Open>03declined Almost every jobs are now running on Nodepool instances w... [07:52:05] (03PS2) 10Addshore: docker: build.sh allow specifying a single image to build [integration/config] - 10https://gerrit.wikimedia.org/r/377314 [07:52:12] (03PS3) 10Addshore: docker: ops-puppet use git cache for cache-buster instead of time [integration/config] - 10https://gerrit.wikimedia.org/r/377320 [07:52:15] (03PS9) 10Addshore: docker: mediawiki-extensions-phan image [integration/config] - 10https://gerrit.wikimedia.org/r/371708 [07:52:19] (03CR) 10jerkins-bot: [V: 04-1] docker: build.sh allow specifying a single image to build [integration/config] - 10https://gerrit.wikimedia.org/r/377314 (owner: 10Addshore) [07:52:26] spam [07:52:29] (03CR) 10jerkins-bot: [V: 04-1] docker: ops-puppet use git cache for cache-buster instead of time [integration/config] - 10https://gerrit.wikimedia.org/r/377320 (owner: 10Addshore) [07:52:32] (03CR) 10jerkins-bot: [V: 04-1] docker: mediawiki-extensions-phan image [integration/config] - 10https://gerrit.wikimedia.org/r/371708 (owner: 10Addshore) [07:53:28] (03PS3) 10Addshore: docker: build.sh allow specifying a single image to build [integration/config] - 10https://gerrit.wikimedia.org/r/377314 [07:53:34] (03PS4) 10Addshore: docker: ops-puppet use git cache for cache-buster instead of time [integration/config] - 10https://gerrit.wikimedia.org/r/377320 [07:53:39] (03PS10) 10Addshore: docker: mediawiki-extensions-phan image [integration/config] - 10https://gerrit.wikimedia.org/r/371708 [07:53:42] (03CR) 10jerkins-bot: [V: 04-1] docker: build.sh allow specifying a single image to build [integration/config] - 10https://gerrit.wikimedia.org/r/377314 (owner: 10Addshore) [07:53:44] addshore: have you tried out the tox --notest patch ( docker: use tox --notest when populating cache ) ? I can just +2 it [07:53:49] yep [07:53:50] that is when less patch to rebase [07:53:50] (03CR) 10jerkins-bot: [V: 04-1] docker: ops-puppet use git cache for cache-buster instead of time [integration/config] - 10https://gerrit.wikimedia.org/r/377320 (owner: 10Addshore) [07:53:55] (03CR) 10jerkins-bot: [V: 04-1] docker: mediawiki-extensions-phan image [integration/config] - 10https://gerrit.wikimedia.org/r/371708 (owner: 10Addshore) [07:54:07] hashar: well, if you review up to *find the commit* [07:54:26] addshore: or dont you have CR+2 rights on the repo to +2 https://gerrit.wikimedia.org/r/#/c/369605/ ?:) [07:54:27] https://gerrit.wikimedia.org/r/#/c/374507 then i can switch to using the new image i build [07:55:05] If you look at and +2 https://gerrit.wikimedia.org/r/#/c/374507 then I'll +2 yours and then update the tag of the build we use in CI :) [07:56:23] https://gerrit.wikimedia.org/r/#/c/374507/8 I am pretty sure we need a full clone [07:56:28] eg when one send several patches in a row [07:56:39] zuul-merger merge the tip of the chain of patch against production [07:56:54] and shallow clone/fetch would eventually miss objects [07:57:36] hmm, but the job doesnt use zuul merger [07:58:32] and isnt ops/puppet ffwd only? [07:59:29] maybe I should wokr on my zuul stuff first https://gerrit.wikimedia.org/r/#/c/375834/ [08:02:03] RECOVERY - Free space - all mounts on deployment-kafka01 is OK: OK: All targets OK [08:02:22] We have 30-fold increase in number of errored jobs :) https://grafana.wikimedia.org/dashboard/db/job-queue-health?refresh=1m&orgId=1&from=now-12h&to=now [08:02:23] have fun [08:02:32] 10Deployment-Systems, 10Scap (Scap3-Adoption-Phase1), 10scap2, 10Wikimedia-IEG-grant-review, and 2 others: Deploy iegreview with scap3 - https://phabricator.wikimedia.org/T129154#2096664 (10MoritzMuehlenhoff) This can be closed? [08:04:00] ooooh [08:04:04] (03CR) 10Hashar: [C: 04-1] "The shallow clone causes troubles when fetching a chain of patches from zuul-merger. Some objects would end up being missing :(" (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/374507 (owner: 10Addshore) [08:04:20] addshore: yeah it is fast forward only to get them merged [08:04:29] addshore: but zuul-merger still craft a merge commit iirc [08:04:37] but really, you end up missing objects [08:04:46] what objects? [08:05:04] the code being checked out will still be exactly the same as if you didnt do a shallow clone / fetch [08:08:01] ooh its a big comment *reads* [08:12:23] back to normal :D [08:13:43] (03CR) 10Addshore: docker: puppet, shallow fetches & no second clone (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/374507 (owner: 10Addshore) [08:15:54] (03PS7) 10Addshore: docker: Tag images when building with a date stamp [integration/config] - 10https://gerrit.wikimedia.org/r/377249 [08:16:26] (03PS5) 10Addshore: docker: ops-puppet use git cache for cache-buster instead of time [integration/config] - 10https://gerrit.wikimedia.org/r/377320 [08:16:55] hashar: ^^ I re ordered the chain with your patch in the front and then 2 that make building images easier after, if you could give those a quick review then I'll merge yours and update the job :) [08:17:09] (03CR) 10Addshore: [V: 031] docker: Tag images when building with a date stamp [integration/config] - 10https://gerrit.wikimedia.org/r/377249 (owner: 10Addshore) [08:17:13] (03CR) 10Addshore: [V: 031] docker: ops-puppet use git cache for cache-buster instead of time [integration/config] - 10https://gerrit.wikimedia.org/r/377320 (owner: 10Addshore) [08:17:36] 10Continuous-Integration-Infrastructure (phase-out-trusty), 10Release-Engineering-Team (Kanban), 10Patch-For-Review: Remove trusty from Nodepool and clean out puppet - https://phabricator.wikimedia.org/T175696#3603463 (10hashar) p:05Triage>03Normal [08:18:17] 10Continuous-Integration-Infrastructure (phase-out-trusty), 10Release-Engineering-Team (Kanban), 10Patch-For-Review: Remove trusty from Nodepool and clean out puppet - https://phabricator.wikimedia.org/T175696#3600481 (10hashar) Puppet cleaning is done via: https://gerrit.wikimedia.org/r/377717 //contint: r... [08:19:12] addshore: sure thing [08:19:53] gotta crash out a few more things first [08:19:58] okay! [08:20:05] 10Continuous-Integration-Infrastructure (phase-out-trusty), 10Release-Engineering-Team (Kanban), 10Patch-For-Review: Remove trusty from Nodepool, WMCS, and clean out puppet - https://phabricator.wikimedia.org/T175696#3603466 (10hashar) [08:23:39] (03CR) 10Hashar: [C: 04-1] docker: puppet, shallow fetches & no second clone (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/374507 (owner: 10Addshore) [08:23:59] addshore: the shallow stuff, I gotta look at it way more [08:24:15] ack! :) yeh, we can leave the shallow stuff for now :) [08:24:22] I am 100% sure it ends up causing troubles but I failed to find any task talking about it [08:24:43] In my head I still can't identify / think ofthe problem [08:25:05] date --utc +Y.%m.%d.%H.%M [08:25:05] Y.09.13.08.24 [08:25:07] !!!! [08:25:19] ah I failed :] [08:26:03] * hashar grabs a coffee [08:35:24] Only [A-Za-z0-9_.-] are allowed [08:35:29] grblblbl [08:36:31] $ date -Iminutes --utc [08:36:31] 2017-09-13T08:36+00:00 [08:36:33] bah [08:38:44] (03CR) 10Hashar: docker: Tag images when building with a date stamp (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/377249 (owner: 10Addshore) [08:39:01] (03CR) 10Hashar: [C: 032] docker: Tag images when building with a date stamp [integration/config] - 10https://gerrit.wikimedia.org/r/377249 (owner: 10Addshore) [08:58:52] RECOVERY - Mediawiki Error Rate on graphite-labs is OK: OK: Less than 1.00% above the threshold [1.0] [09:12:23] (03CR) 10Hashar: [C: 04-1] "I like the idea of using the sha1 of the HEAD branch to burst the cache. That makes total sense." (032 comments) [integration/config] - 10https://gerrit.wikimedia.org/r/377320 (owner: 10Addshore) [09:12:32] addshore: https://gerrit.wikimedia.org/r/projects/operations%2Fpuppet/branches/production [09:12:38] is wayy faster than git ls-remote [09:12:45] but maybe it does not matter [09:13:08] so I guess you can at least +2 mine https://gerrit.wikimedia.org/r/#/c/369605/ [09:13:21] and I already CR+2 the patch that tweak the image tagging [09:14:30] !log nodepool: openstack image delete image-ci-trusty - T175696 [09:14:31] cool, was just filling out loads of forms, but back now :) [09:14:34] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [09:14:34] T175696: Remove trusty from Nodepool, WMCS, and clean out puppet - https://phabricator.wikimedia.org/T175696 [09:15:01] GIT_TRACE_PACKET=1 GIT_TRACE=1 git -c uploadpack.hideRefs=refs/notes/review ls-remote https://gerrit.wikimedia.org/r/operations/puppet HEAD [09:15:12] (03CR) 10Addshore: [C: 032] docker: use tox --notest when populating cache [integration/config] - 10https://gerrit.wikimedia.org/r/369605 (owner: 10Hashar) [09:15:13] addshore: try that ^^ ^ that will show all the stuff the server end up sending back to you :( [09:15:21] eg all of ref/changes/** [09:15:37] I'll build an image with your change in now and take a look at that idea as I do! :) [09:16:14] (03Merged) 10jenkins-bot: docker: use tox --notest when populating cache [integration/config] - 10https://gerrit.wikimedia.org/r/369605 (owner: 10Hashar) [09:16:16] (03Merged) 10jenkins-bot: docker: Tag images when building with a date stamp [integration/config] - 10https://gerrit.wikimedia.org/r/377249 (owner: 10Addshore) [09:17:14] also, I think the tags I have of docker images that are not merged I might call prefix with dev instead of v [09:19:05] addshore: +1 [09:19:18] and at some point we will probably want a "stable" tag [09:19:25] run CI out of it [09:19:35] and do some tests against "latest" to latter promote it to "stable" [09:19:38] but that is long term :] [09:22:57] hashar: yeh, that would save bumping the versions in the jenkins jobs by 'hand' [09:23:29] and we will want things like stable-0.8.5 and stuff for 0.8.5 version of phan vs 0.8 etc and things [09:25:16] !log Deleting integration-slave-trusty-1003 and integration-slave-trusty-1001 - T175696 [09:25:20] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [09:25:20] T175696: Remove trusty from Nodepool, WMCS, and clean out puppet - https://phabricator.wikimedia.org/T175696 [09:25:44] addshore: for phan, yes definitely :] [09:28:28] PROBLEM - Host integration-slave-trusty-1001 is DOWN: CRITICAL - Host Unreachable (10.68.16.168) [09:28:46] PROBLEM - Host integration-slave-trusty-1003 is DOWN: CRITICAL - Host Unreachable (10.68.17.54) [09:30:33] (03CR) 10Addshore: "Pushing as v2017.09.13.09.23" [integration/config] - 10https://gerrit.wikimedia.org/r/377249 (owner: 10Addshore) [09:31:28] (03PS1) 10Addshore: docker: use wmfreleng/operations-puppet:v2017.09.13.09.23 [integration/config] - 10https://gerrit.wikimedia.org/r/377720 [09:31:43] (03CR) 10Addshore: [V: 04-1 C: 04-2] "Not finished pushing yet" [integration/config] - 10https://gerrit.wikimedia.org/r/377720 (owner: 10Addshore) [09:40:16] PROBLEM - Puppet errors on integration-slave-docker-1001 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [09:43:03] (03CR) 10Hashar: [C: 031] "And later on, I guess we will want to introduce a 'stable' tag. That will save us from having to update the Jenkins job each time an image" [integration/config] - 10https://gerrit.wikimedia.org/r/377720 (owner: 10Addshore) [09:43:52] (03CR) 10Hashar: [C: 032] "Deployed. I forgot to CR+2 it" [integration/config] - 10https://gerrit.wikimedia.org/r/377712 (https://phabricator.wikimedia.org/T161882) (owner: 10Hashar) [09:44:04] kids && lunch & [09:47:00] (03Merged) 10jenkins-bot: Migrate php-compile-php55 to Jessie [integration/config] - 10https://gerrit.wikimedia.org/r/377712 (https://phabricator.wikimedia.org/T161882) (owner: 10Hashar) [10:13:50] !log docker push docker.io/wmfreleng/operations-puppet:v2017.09.13.09.23 (#d693f74c9b3404220a2ad2934f526d4f4455914b) [10:13:54] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [10:15:18] RECOVERY - Puppet errors on integration-slave-docker-1001 is OK: OK: Less than 1.00% above the threshold [0.0] [10:17:02] (03CR) 10Addshore: docker: use wmfreleng/operations-puppet:v2017.09.13.09.23 [integration/config] - 10https://gerrit.wikimedia.org/r/377720 (owner: 10Addshore) [10:19:41] Hi everybody, has anything changed recently in Jenkins related to Job/Release permissions? [10:20:00] me and the analytics team are getting an error while doing https://integration.wikimedia.org/ci/job/analytics-refinery-release/m2release/ (that is our maven release of the refinery jars) [10:20:10] "Elukey is missing the Job/Release permission" [10:21:38] (03CR) 10Addshore: [C: 032] docker: use wmfreleng/operations-puppet:v2017.09.13.09.23 [integration/config] - 10https://gerrit.wikimedia.org/r/377720 (owner: 10Addshore) [10:21:59] (03CR) 10Addshore: [C: 032] "Deployed & checked in https://integration.wikimedia.org/ci/job/operations-puppet-tests-docker/5125/console using https://gerrit.wikimedia." [integration/config] - 10https://gerrit.wikimedia.org/r/377720 (owner: 10Addshore) [10:22:14] elukey: oooooh [10:22:21] elukey: let me find the ticket [10:22:40] https://phabricator.wikimedia.org/T169557 [10:22:45] * elukey hugs addshore [10:22:59] I guess your not in the ciadmin group? [10:23:19] I also guess nda/wmde/wmf used to have Release [10:24:31] yep my colleagues were able to release [10:24:50] buuut we haven't been doing deployments in a while :D [10:24:55] (03Merged) 10jenkins-bot: docker: use wmfreleng/operations-puppet:v2017.09.13.09.23 [integration/config] - 10https://gerrit.wikimedia.org/r/377720 (owner: 10Addshore) [10:28:27] thanks addshore ! [10:33:42] 10Release-Engineering-Team (Next), 10Release Pipeline: Find CI container build location - https://phabricator.wikimedia.org/T173128#3603819 (10elukey) [10:52:30] PROBLEM - Puppet errors on deployment-imagescaler02 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [10:54:47] PROBLEM - Puppet errors on deployment-urldownloader is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [11:29:41] 10Release-Engineering-Team, 10Operations, 10TCB-Team, 10WMDE-QWERTY-Team-Board: Deploy new Wikidiff2 version on beta-cluster - https://phabricator.wikimedia.org/T175818#3604022 (10Tobi_WMDE_SW) @Legoktm @greg, adding the teams #operations and #release-engineering-team as I'm not sure who exactly would be i... [11:29:48] RECOVERY - Puppet errors on deployment-urldownloader is OK: OK: Less than 1.00% above the threshold [0.0] [11:32:31] RECOVERY - Puppet errors on deployment-imagescaler02 is OK: OK: Less than 1.00% above the threshold [0.0] [11:34:14] 10Release-Engineering-Team, 10Operations, 10TCB-Team, 10User-Addshore, 10WMDE-QWERTY-Team-Board: Deploy new Wikidiff2 version on beta-cluster - https://phabricator.wikimedia.org/T175818#3604031 (10Addshore) [11:52:22] 10Release-Engineering-Team, 10Operations, 10TCB-Team, 10User-Addshore, 10WMDE-QWERTY-Team-Board: Deploy new Wikidiff2 version on beta-cluster - https://phabricator.wikimedia.org/T175818#3603917 (10MoritzMuehlenhoff) We need to update the package on apt.wikimedia.org, then it's available on the beta clust... [11:53:02] 10Release-Engineering-Team, 10Operations, 10TCB-Team, 10User-Addshore, 10WMDE-QWERTY-Team-Board: Deploy new Wikidiff2 version on beta-cluster - https://phabricator.wikimedia.org/T175818#3604072 (10MoritzMuehlenhoff) a:03MoritzMuehlenhoff [12:00:03] 10Release-Engineering-Team, 10Operations, 10TCB-Team, 10User-Addshore, 10WMDE-QWERTY-Team-Board: Deploy new Wikidiff2 version on beta-cluster - https://phabricator.wikimedia.org/T175818#3604077 (10Addshore) I believe you need both 410ab2ff636eed296206b80a3c89aa75a50b0f8a and a1d711ebb5ce6bde66b2e4b1e6503... [12:45:59] 10Continuous-Integration-Infrastructure, 10Patch-For-Review: integration-slave-jessie-1003 (and others?) missing jsduck executable - https://phabricator.wikimedia.org/T175764#3604164 (10hashar) Looks like it has been manually installed on one of the slave which is the one I used to verify whether jsduck is pro... [12:49:52] 10Continuous-Integration-Infrastructure, 10Patch-For-Review: integration-slave-jessie-1003 (and others?) missing jsduck executable - https://phabricator.wikimedia.org/T175764#3604173 (10hashar) Should be good now: ```integration-slave-jessie-1003.integration.eqiad.wmflabs: JSDuck 5.3.4 (Ruby 2.1.5) integr... [12:50:54] funny the puppet docker job fails when it has to rebuild the image https://integration.wikimedia.org/ci/job/operations-puppet-tests-docker/5139/console [12:50:56] addshore: ^^ [12:50:59] I am off again [13:09:40] hasharAway: ahh, build timed out, the initial pull took too long [13:26:08] zeljkof: what can I do to push forward with https://phabricator.wikimedia.org/T167432 ? [13:27:48] addshore: rewrite the tests in node.js? ;) [13:28:18] sorry, I was really busy with selenium in node lately, this is on my list [13:29:43] I will try to make some progress this week, is that ok? sorry, but I can not estimate when I will be able to resolve it [13:30:33] 10Browser-Tests-Infrastructure, 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban), 10Wikidata, and 3 others: Run Wikibase daily browser tests on Jenkins - https://phabricator.wikimedia.org/T167432#3332825 (10zeljkofilipin) a:03zeljkofilipin [13:31:06] 10Browser-Tests-Infrastructure, 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban), 10Wikidata, and 3 others: Run Wikibase daily browser tests on Jenkins - https://phabricator.wikimedia.org/T167432#3332825 (10zeljkofilipin) I am really busy with Selenium in Node.js lately. I will... [13:46:09] zeljkof: okay! [13:46:32] zeljkof: is there not already a job that runs on every commit that we can just copy for now to run daily? (with whatever extra is wanted) ? [13:47:01] addshore: not sure, I will check, but probably not today [13:48:50] 10Browser-Tests-Infrastructure, 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban), 10Wikidata, and 3 others: Run Wikibase daily browser tests on Jenkins - https://phabricator.wikimedia.org/T167432#3604380 (10Addshore) [13:50:34] zeljkof: okay! I might poke it a little bit today [13:51:09] there should not be anything complicated, I was just busy with other stuff :( [13:51:15] ack! :) [13:52:04] Yeh and I mean, https://integration.wikimedia.org/ci/job/mwext-mw-selenium-composer-jessie/5988/ works on each commit so I might be able to figure it out [14:14:40] Yippee, build fixed! [14:14:40] Project selenium-MinervaNeue » chrome,beta,Linux,BrowserTests build #116: 09FIXED in 19 min: https://integration.wikimedia.org/ci/job/selenium-MinervaNeue/BROWSER=chrome,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=BrowserTests/116/ [14:28:38] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team (Next), 10Cloud-Services, 10Puppet, 10User-Joe: Re-think puppet management for deployment-prep - https://phabricator.wikimedia.org/T161675#3604507 (10chasemp) p:05Triage>03Normal [14:57:37] 10Browser-Tests-Infrastructure, 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban), 10Wikidata, and 3 others: Run Wikibase daily browser tests on Jenkins - https://phabricator.wikimedia.org/T167432#3604612 (10Tobi_WMDE_SW) [15:17:05] 10Release-Engineering-Team (Kanban), 10Release, 10Train Deployments: 1.30.0-wmf.18 deployment blockers - https://phabricator.wikimedia.org/T170636#3604694 (10demon) [15:18:42] 10Gerrit-Migration, 10Release-Engineering-Team (Backlog), 10Differential, 10Phabricator, and 2 others: Create conduit method to query the feed and return records with relevant details populated instead of just a bunch of phids - https://phabricator.wikimedia.org/T123417#3604696 (10Aklapper) > there aren't... [15:20:17] 10Continuous-Integration-Infrastructure, 10Patch-For-Review: integration-slave-jessie-1003 (and others?) missing jsduck executable - https://phabricator.wikimedia.org/T175764#3604715 (10Jdforrester-WMF) 05Open>03Resolved a:03hashar Yup, all looks good. Thanks! https://integration.wikimedia.org/ci/job/mwe... [15:27:28] PROBLEM - Puppet errors on deployment-prometheus01 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [15:47:15] 10Scap (Scap3-Adoption-Phase1), 10releng-201516-q4, 10releng-201718-q1, 10Trebuchet: [keyresult] Migrate remaining trebuchet deployed services - https://phabricator.wikimedia.org/T129290#3604802 (10thcipriani) [15:47:18] 10Deployment-Systems, 10Scap (Scap3-Adoption-Phase1), 10scap2, 10Wikimedia-IEG-grant-review, and 2 others: Deploy iegreview with scap3 - https://phabricator.wikimedia.org/T129154#3604799 (10thcipriani) 05Open>03Resolved a:03thcipriani deployed! [15:47:27] 10Scap (Scap3-Adoption-Phase1), 10releng-201516-q4, 10releng-201718-q1, 10Trebuchet: [keyresult] Migrate remaining trebuchet deployed services - https://phabricator.wikimedia.org/T129290#2101252 (10thcipriani) [15:47:29] 10Scap (Scap3-Adoption-Phase1), 10Wikimedia-Wikimania-Scholarships, 10Patch-For-Review, 10User-bd808: Deploy scholarships with scap3 - https://phabricator.wikimedia.org/T129134#3604803 (10thcipriani) 05Open>03Resolved Deployed! [15:50:01] (03PS4) 10Addshore: docker: build.sh allow specifying a single image to build [integration/config] - 10https://gerrit.wikimedia.org/r/377314 [15:51:10] PROBLEM - Long lived cherry-picks on puppetmaster on deployment-puppetmaster02 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [15:55:36] (03PS11) 10Addshore: docker: mediawiki-extensions-phan image [integration/config] - 10https://gerrit.wikimedia.org/r/371708 [16:02:27] RECOVERY - Puppet errors on deployment-prometheus01 is OK: OK: Less than 1.00% above the threshold [0.0] [16:04:55] 10Continuous-Integration-Infrastructure (phase-out-trusty), 10Release-Engineering-Team (Kanban), 10Patch-For-Review: Package php modules for Zend 5.5 on Jessie - https://phabricator.wikimedia.org/T174972#3578458 (10Jdforrester-WMF) >>! In T174972#3600734, @hashar wrote: > https://gerrit.wikimedia.org/r/#/c/3... [17:01:53] 10Scap (Scap3-Adoption-Phase1), 10Wikimedia-Wikimania-Scholarships, 10Patch-For-Review, 10User-bd808: Deploy scholarships with scap3 - https://phabricator.wikimedia.org/T129134#2096320 (10Niharika) Thanks all! [17:22:22] 10Gerrit-Migration, 10Release-Engineering-Team (Backlog), 10Differential, 10Phabricator, and 2 others: Create conduit method to query the feed and return records with relevant details populated instead of just a bunch of phids - https://phabricator.wikimedia.org/T123417#3605099 (10ksmith) @JAufrecht : Perh... [17:27:30] 10Release-Engineering-Team, 10Operations, 10TCB-Team, 10User-Addshore, 10WMDE-QWERTY-Team-Board: Deploy new Wikidiff2 version on beta-cluster - https://phabricator.wikimedia.org/T175818#3605122 (10MaxSem) This should definitely be a new version, 1.5. [17:28:00] 10Continuous-Integration-Config, 10MinusX: Add MinusX to MediaWiki extensions and other repositories - https://phabricator.wikimedia.org/T175794#3603339 (10demon) I don't think it can be added en-masse to non-MediaWiki repositories. Too many false positives that we'll then require users to exempt from? I'm th... [17:31:00] 10Continuous-Integration-Config, 10MinusX: Add MinusX to MediaWiki extensions and other repositories - https://phabricator.wikimedia.org/T175794#3605144 (10Legoktm) Right, that makes sense. I was mostly thinking about PHP library repos at the time, let me reword it to mean that. [17:31:21] 10Continuous-Integration-Config, 10MinusX: Add MinusX to MediaWiki extensions and PHP library repos - https://phabricator.wikimedia.org/T175794#3605145 (10Legoktm) [17:47:09] 10Deployment-Systems, 10Scap (Scap3-Adoption-Phase1), 10scap2, 10Discovery: Deploy discovery-analytics with scap3 - https://phabricator.wikimedia.org/T129149#2096588 (10ksmith) This was mentioned in Scrum-of-scrums. Who would actually do the work to get it migrated? [17:47:25] 10Gerrit-Migration, 10Release-Engineering-Team (Backlog), 10Differential, 10Phabricator, and 2 others: Create conduit method to query the feed and return records with relevant details populated instead of just a bunch of phids - https://phabricator.wikimedia.org/T123417#3605210 (10JAufrecht) In the long ru... [17:48:58] 10Release-Engineering-Team (Kanban), 10Release Pipeline (Blubber): Package Blubber - https://phabricator.wikimedia.org/T175609#3605211 (10dduvall) [17:51:31] 10Gerrit-Migration, 10Release-Engineering-Team (Backlog), 10Differential, 10Phabricator, and 2 others: Create conduit method to query the feed and return records with relevant details populated instead of just a bunch of phids - https://phabricator.wikimedia.org/T123417#1929298 (10greg) "Conduit" is just t... [17:53:39] (03CR) 10Thcipriani: [C: 032] "Works as expected." (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/377314 (owner: 10Addshore) [17:54:36] (03Merged) 10jenkins-bot: docker: build.sh allow specifying a single image to build [integration/config] - 10https://gerrit.wikimedia.org/r/377314 (owner: 10Addshore) [17:56:03] twentyafterfour: I can help with OCG testing; someone from Reading Web can probably help too [17:56:21] I'm still hoping to get rid of OCG by EOQ though [17:56:48] tgr: if it's going away then maybe we don't need to convert it to scap deploy? [17:56:58] I think EOQ is the deadline for trebuchet going away [17:57:10] thcipriani: ^ [17:57:31] when do you need to know for sure? [17:58:10] we are deploying the replacement code this week, will need to run some tests to see if electron can handle the extra load [17:58:24] ah, nice. [17:58:31] so we are pretty close but EOQ is also pretty close... [17:59:54] I think I will go ahead and write some patches for OCG, but we won't schedule a migration unless it looks like replacement code isn't going to work out. [18:00:47] but it's tricky because ops also needs time to remove salt. [18:17:55] 10Release-Engineering-Team, 10Operations, 10TCB-Team, 10User-Addshore, 10WMDE-QWERTY-Team-Board: Deploy new Wikidiff2 version on beta-cluster - https://phabricator.wikimedia.org/T175818#3605304 (10Legoktm) I tagged `1.5.0`, and prepared packaging changes: * Merge tag '1.5.0' into debian - https://gerrit... [18:21:34] 10Continuous-Integration-Config, 10MinusX: Add MinusX to MediaWiki extensions and PHP library repos - https://phabricator.wikimedia.org/T175794#3603339 (10Mainframe98) (User story) As an extension maintainer, I would like to add this to the extension(s) I maintain. How would I go about doing that? Is there goi... [18:31:26] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team (Watching / External), 10Operations, 10TCB-Team, and 2 others: Deploy new Wikidiff2 version on beta-cluster - https://phabricator.wikimedia.org/T175818#3605370 (10greg) [18:43:57] Thanks for the merge thcipriani! [18:44:18] addshore: thanks for the patch, good feature :) [18:44:34] * addshore is currently sat on an airport floor waiting for a flight that he doesnt even have a seat for right now as it is overbooked.... [18:44:50] that sounds like a bummer [18:51:57] addshore: be the co-pilot xD [18:52:11] 10Release-Engineering-Team (Watching / External), 10Phlogiston (Requests): Adjust phlogiston configuration for Release Engineering - https://phabricator.wikimedia.org/T170359#3605525 (10JAufrecht) [18:58:51] 10Continuous-Integration-Config, 10MinusX: Add MinusX to MediaWiki extensions and PHP library repos - https://phabricator.wikimedia.org/T175794#3603339 (10Tgr) >>! In T175794#3605327, @Mainframe98 wrote: > Also, while I can infer the steps I need to take from the linked patch it still needs some documentation... [19:00:01] 10Continuous-Integration-Infrastructure (phase-out-trusty), 10Release-Engineering-Team (Kanban), 10Patch-For-Review: Package php modules for Zend 5.5 on Jessie - https://phabricator.wikimedia.org/T174972#3605578 (10hashar) Moritz is rebuilding the Debian packages and will published them on apt.wikimedia.org... [19:05:40] (03PS6) 10Addshore: docker: ops-puppet use git cache for cache-buster instead of time [integration/config] - 10https://gerrit.wikimedia.org/r/377320 [19:05:50] (03PS7) 10Addshore: docker: ops-puppet use git hash for cache-buster instead of time [integration/config] - 10https://gerrit.wikimedia.org/r/377320 [19:06:31] thcipriani: ^^ thats another cool thing to speed up builds when nothing in the puppet repo has changed :) [19:07:04] ah, nice :) [19:18:53] grbmbm [19:19:29] Step 0 : FROM docker-registry.wikimedia.org/wikimedia-jessie:latest as builder [19:19:30] Pulling repository docker-registry.wikimedia.org/wikimedia-jessie [19:19:40] thcipriani: addshore: ^^ any clue why docker screams at me ? :] [19:19:51] INFO[0001] Error: image wikimedia-jessie:latest as builder not found [19:20:06] ohwrong docker version i guess [19:20:21] yeh! thatll be it! [19:20:32] :) [19:21:00] cant we just use debootstrap / chroot to build a tarball [19:21:10] and lxc as a runner? :]]]]]]]]]]]]]]]]] [19:21:23] * addshore throws tomatoes at hashar [19:40:45] Error response from daemon: client and server don't have same version (client : 1.24, server: 1.18) [19:40:45] progress! [19:48:45] bah debian has 1.13 which is not enough :D [19:50:55] will upgrade tomorrow ! [20:35:00] Is wmf.23 (cut on 2017-10-17) going to be the last cut of 1.30 or will it continue? [20:37:24] no_justification: ^ :) :) [20:38:56] No idea [20:39:03] 10Continuous-Integration-Infrastructure, 10DNS, 10Traffic: CI: operations-dns-lint broken due to missing Maxmind DB file - https://phabricator.wikimedia.org/T175864#3605883 (10Dzahn) [20:39:29] 10Continuous-Integration-Infrastructure, 10DNS, 10Operations, 10Traffic: CI: operations-dns-lint broken due to missing Maxmind DB file - https://phabricator.wikimedia.org/T175864#3605898 (10Dzahn) [20:39:48] 10Continuous-Integration-Infrastructure, 10DNS, 10Operations, 10Traffic: CI: operations-dns-lint broken due to missing Maxmind DB file - https://phabricator.wikimedia.org/T175864#3605902 (10Dzahn) [20:40:32] 10Release-Engineering-Team (Watching / External), 10Operations, 10ops-eqdfw: setup/install/deploy deploy1001 as deployment server - https://phabricator.wikimedia.org/T175288#3605904 (10Cmjohnson) Swapped the motherboard, the error still presented during installation. It's looking more like something else oth... [20:40:49] 10Continuous-Integration-Infrastructure, 10DNS, 10Operations, 10Traffic: CI: operations-dns-lint broken due to missing Maxmind DB file - https://phabricator.wikimedia.org/T175864#3605883 (10Dzahn) [20:42:54] Project selenium-Echo » chrome,beta,Linux,BrowserTests build #516: 04FAILURE in 1 min 52 sec: https://integration.wikimedia.org/ci/job/selenium-Echo/BROWSER=chrome,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=BrowserTests/516/ [20:42:54] Project selenium-Echo » firefox,beta,Linux,BrowserTests build #516: 04FAILURE in 1 min 53 sec: https://integration.wikimedia.org/ci/job/selenium-Echo/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=BrowserTests/516/ [20:46:41] twentyafterfour: this https://gerrit.wikimedia.org/r/#/c/374054/ and https://gerrit.wikimedia.org/r/#/c/354247/ to be merged tomorrow in the maintenance window? would that be good? [20:46:49] saw it on calendar [20:47:43] and for https://gerrit.wikimedia.org/r/#/c/370622/ am i totally derailing it by suggesting systemctl instead of service? i think we should, but i could also amend? [20:48:21] no_justification: James_F, if habit continues, that's about right. Next week being the cut over and then the release about a month later in late Oct [20:48:44] mutante: go ahead and amend sure [20:49:08] greg-g: Right-o. I won't create the wmf.24 milestone then. :-) [20:50:08] Or "milesphone"? "Philestone"? Surely Phab has a terrible name for it… [20:50:33] 10Release-Engineering-Team (Kanban), 10Release, 10Train Deployments: 1.30.0-wmf.24 deployment blockers - https://phabricator.wikimedia.org/T174361#3605939 (10greg) This should probably be 1.31-wmf.1 (and rename the rest in the series) if we plan to release 1.30 late October (to give us a month after REL cut). [20:50:58] kidney stone? [20:50:59] no [20:51:02] different [20:52:43] twentyafterfour: ok :) [21:06:19] PROBLEM - Puppet errors on integration-slave-docker-1001 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [21:12:21] 10Release-Engineering-Team (Kanban), 10Phabricator: Custom task form for #WMF-CTO-Team-Backlog - https://phabricator.wikimedia.org/T175869#3606030 (10JAufrecht) [21:13:57] 10Release-Engineering-Team (Kanban), 10Phabricator: Custom task form for #WMF-CTO-Team-Backlog - https://phabricator.wikimedia.org/T175869#3606046 (10JAufrecht) [21:14:28] 10Release-Engineering-Team (Kanban), 10Phabricator: Custom task form for #WMF-CTO-Team-Backlog - https://phabricator.wikimedia.org/T175869#3606030 (10JAufrecht) [21:15:13] 10Release-Engineering-Team (Kanban), 10Phabricator: Custom task form for #WMF-CTO-Team-Backlog - https://phabricator.wikimedia.org/T175869#3606030 (10JAufrecht) [21:17:20] (03PS4) 10Umherirrender: Add missing unit test, npm jobs and make tests voting [integration/config] - 10https://gerrit.wikimedia.org/r/376761 [21:32:10] (03CR) 10jerkins-bot: [V: 04-1] Add missing unit test, npm jobs and make tests voting [integration/config] - 10https://gerrit.wikimedia.org/r/376761 (owner: 10Umherirrender) [21:41:25] 10Release-Engineering-Team (Watching / External), 10Wikidata: Decide what to do with Wikibase JS-only libraries regarding the build/deployment of Wikidata code - https://phabricator.wikimedia.org/T174922#3606072 (10Legoktm) I like Krinkle's idea the most (option #5) for now, mostly because it's probably the ea... [21:52:09] 10Gerrit-Migration, 10Release-Engineering-Team (Kanban), 10Differential, 10Phabricator, and 2 others: Create conduit method to query the feed and return records with relevant details populated instead of just a bunch of phids - https://phabricator.wikimedia.org/T123417#3606090 (10mmodell) a:03mmodell [21:52:21] 10Gerrit-Migration, 10Release-Engineering-Team (Kanban), 10Differential, 10Phabricator, and 2 others: Create conduit method to query the feed and return records with relevant details populated instead of just a bunch of phids - https://phabricator.wikimedia.org/T123417#3606107 (10mmodell) 05stalled>03Op... [21:52:55] 10Gerrit-Migration, 10Differential, 10Wikibugs: Broadcast Differential activity to IRC - https://phabricator.wikimedia.org/T116330#3606110 (10mmodell) 05stalled>03Open [21:53:04] 10Gerrit-Migration, 10Release-Engineering-Team (Kanban), 10Differential, 10Wikibugs: Broadcast Differential activity to IRC - https://phabricator.wikimedia.org/T116330#3606112 (10mmodell) a:03mmodell [22:16:08] 10Gerrit-Migration, 10Release-Engineering-Team (Kanban), 10Differential, 10Phabricator, and 2 others: Create conduit method to query the feed and return records with relevant details populated instead of just a bunch of phids - https://phabricator.wikimedia.org/T123417#3606199 (10JAufrecht) Added {T175872}... [22:16:18] RECOVERY - Puppet errors on integration-slave-docker-1001 is OK: OK: Less than 1.00% above the threshold [0.0] [22:19:01] 10Release-Engineering-Team (Watching / External), 10Phlogiston (Requests): Adjust phlogiston configuration for Release Engineering - https://phabricator.wikimedia.org/T170359#3606210 (10JAufrecht) a:03JAufrecht [22:42:16] PROBLEM - Puppet errors on integration-slave-docker-1001 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [23:17:18] RECOVERY - Puppet errors on integration-slave-docker-1001 is OK: OK: Less than 1.00% above the threshold [0.0] [23:18:45] 10Deployment-Systems, 10Scap (Scap3-Adoption-Phase1), 10scap2, 10Discovery, 10Patch-For-Review: Deploy discovery-analytics with scap3 - https://phabricator.wikimedia.org/T129149#3606382 (10thcipriani) >>! In T129149#3605208, @ksmith wrote: > This was mentioned in Scrum-of-scrums. Who would actually do th... [23:38:18] PROBLEM - Puppet errors on integration-slave-docker-1001 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0]