[00:00:26] PROBLEM - Puppet errors on deployment-tin is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [00:23:16] (03CR) 10Thcipriani: Add --update-jjb argument to build.py (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/382341 (owner: 10Legoktm) [00:23:49] oh heh [00:24:30] yeah that should work I think. [00:26:42] nope, not working [00:29:46] oh I'm stupid. I've been running the wrong script [00:29:59] (03CR) 10Legoktm: Add --update-jjb argument to build.py (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/382341 (owner: 10Legoktm) [00:30:26] RECOVERY - Puppet errors on deployment-tin is OK: OK: Less than 1.00% above the threshold [0.0] [00:30:47] (03PS5) 10Legoktm: Add --update-jjb argument to build.py [integration/config] - 10https://gerrit.wikimedia.org/r/382341 [00:30:49] (03PS5) 10Legoktm: build.py: Improve "docker push" output [integration/config] - 10https://gerrit.wikimedia.org/r/382347 [00:30:51] (03PS8) 10Legoktm: docker: Recursively build images in order [integration/config] - 10https://gerrit.wikimedia.org/r/381761 (https://phabricator.wikimedia.org/T176318) (owner: 10Hashar) [00:32:22] there is likely a non-zero number of deaths caused my regexes in a given year :) [00:32:32] xP [00:32:57] 10Gerrit, 10UI-Standardization: Make gerrit colors align with WikimediaUI color palette - https://phabricator.wikimedia.org/T158298#3663168 (10Volker_E) [00:34:20] (03CR) 10Thcipriani: [C: 032] Add --update-jjb argument to build.py [integration/config] - 10https://gerrit.wikimedia.org/r/382341 (owner: 10Legoktm) [00:35:23] (03Merged) 10jenkins-bot: Add --update-jjb argument to build.py [integration/config] - 10https://gerrit.wikimedia.org/r/382341 (owner: 10Legoktm) [00:35:25] (03CR) 10Thcipriani: [C: 032] build.py: Improve "docker push" output [integration/config] - 10https://gerrit.wikimedia.org/r/382347 (owner: 10Legoktm) [00:36:16] still haven't reviewed the recursive build one yet. [00:36:54] (03Merged) 10jenkins-bot: build.py: Improve "docker push" output [integration/config] - 10https://gerrit.wikimedia.org/r/382347 (owner: 10Legoktm) [00:47:38] thcipriani|afk: no worries, it's a lot more complicated :p [03:53:10] 10RelEng-Archive-FY201718-Q1, 10Release, 10Train Deployments: 1.31.0-wmf.1 deployment blockers - https://phabricator.wikimedia.org/T172806#3663341 (10Liuxinyu970226) [04:38:02] PROBLEM - Free space - all mounts on deployment-fluorine02 is CRITICAL: CRITICAL: deployment-prep.deployment-fluorine02.diskspace._srv.byte_percentfree (<30.00%) [05:52:45] (03PS1) 10Legoktm: Add jobs for mediawiki/libs/Equivset [integration/config] - 10https://gerrit.wikimedia.org/r/382645 [05:57:28] (03CR) 10Legoktm: [C: 032] Add jobs for mediawiki/libs/Equivset [integration/config] - 10https://gerrit.wikimedia.org/r/382645 (owner: 10Legoktm) [05:59:05] (03Merged) 10jenkins-bot: Add jobs for mediawiki/libs/Equivset [integration/config] - 10https://gerrit.wikimedia.org/r/382645 (owner: 10Legoktm) [06:39:48] PROBLEM - Mediawiki Error Rate on graphite-labs is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [10.0] [06:55:44] 10Gerrit, 10Release-Engineering-Team (Backlog), 10DBA, 10Operations, 10Patch-For-Review: Gerrit shows HTTP 500 error when pasting extended unicode characters - https://phabricator.wikimedia.org/T145885#3663481 (10Marostegui) >>! In T145885#3555784, @Paladox wrote: > I doint think this is worth it now sin... [07:03:01] RECOVERY - Free space - all mounts on deployment-fluorine02 is OK: OK: All targets OK [07:39:51] PROBLEM - Mediawiki Error Rate on graphite-labs is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [10.0] [08:23:46] 10Continuous-Integration-Config, 10Release-Engineering-Team, 10Composer: Composer failed in Selenium job but job didn't stop - https://phabricator.wikimedia.org/T177047#3663689 (10hashar) The shell snippet being run is: ``` lang=bash set -u cd "src" [[ -f "composer.json" ]] || exit 0 composer update --ansi -... [08:27:02] 10Continuous-Integration-Infrastructure (shipyard), 10Release-Engineering-Team: On CI, upgrade docker-ce from 17.06.2 to 17.09.0 - https://phabricator.wikimedia.org/T177499#3663703 (10hashar) p:05Triage>03Normal The memory leak I pointed at ( T177039 ) was most probably due to an outdated Linux Kernel. Doc... [08:28:01] 10Continuous-Integration-Infrastructure (shipyard), 10Release-Engineering-Team (Kanban): mediawiki-core-phpcs-docker jobs running on integration-slave-docker-1001 are running significantly slower than other slaves - https://phabricator.wikimedia.org/T177039#3663706 (10hashar) [08:29:08] 10Continuous-Integration-Infrastructure (shipyard), 10Release-Engineering-Team (Kanban): mediawiki-core-phpcs-docker jobs running on integration-slave-docker-1001 are running significantly slower than other slaves - https://phabricator.wikimedia.org/T177039#3645351 (10hashar) p:05Triage>03Normal Lets monit... [08:29:16] 10Continuous-Integration-Infrastructure (shipyard), 10Release-Engineering-Team (Kanban): mediawiki-core-phpcs-docker jobs running on integration-slave-docker-1001 are running significantly slower than other slaves - https://phabricator.wikimedia.org/T177039#3663722 (10hashar) a:03hashar [08:45:54] 10Continuous-Integration-Infrastructure (shipyard): CI docker build should use an apt cache - https://phabricator.wikimedia.org/T175966#3663738 (10hashar) 05Open>03Resolved a:03hashar `docker build` does not support mounting a volume. There is a RedHat patch to do it but that only accepts a read-only volum... [08:46:26] 10Continuous-Integration-Infrastructure (shipyard), 10Release-Engineering-Team (Kanban): CI docker build should use an apt cache - https://phabricator.wikimedia.org/T175966#3663742 (10hashar) [08:47:27] addshore: if you are around. The docker slave had some bad memory usage. Seems like that was due to some old Linux kernel and it should be fine now :] [08:48:16] hashar: nice [08:48:31] are any other of the slaves on ye olde kernel? [08:48:45] addshore: your guess? :D [08:48:50] I got them upgraded yesterday [08:48:57] wheee :) [08:49:08] tyler eventually figured out the faulty slave docker-1002 had an old kernel [08:49:19] docker-1001 got rebooted like 7 days ago and apparently no more leak [08:49:38] we will see. I will mark the task resolved end of next week I guess [08:49:48] RECOVERY - Mediawiki Error Rate on graphite-labs is OK: OK: Less than 1.00% above the threshold [1.0] [09:22:35] !log integration: purged bunch of old containers: sudo cumin 'name:slave-docker' 'yes | docker container prune' [09:22:40] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [09:36:16] 10Gerrit, 10Release-Engineering-Team (Backlog), 10DBA, 10Operations, 10Patch-For-Review: Gerrit shows HTTP 500 error when pasting extended unicode characters - https://phabricator.wikimedia.org/T145885#3663819 (10Paladox) >>! In T145885#3663481, @Marostegui wrote: >>>! In T145885#3555784, @Paladox wrote:... [09:45:31] 10Gerrit, 10Release-Engineering-Team (Backlog), 10DBA, 10Operations, 10Patch-For-Review: Gerrit shows HTTP 500 error when pasting extended unicode characters - https://phabricator.wikimedia.org/T145885#3663883 (10Paladox) I guess we can remove DBA now as there is nothing here now for the DBA to do? [09:50:48] (03CR) 10Hashar: "Nice :)" [integration/config] - 10https://gerrit.wikimedia.org/r/381761 (https://phabricator.wikimedia.org/T176318) (owner: 10Hashar) [11:18:29] hashar: Hey! I'm modernizing and factorizing our maven builds. (https://gerrit.wikimedia.org/r/#/c/381763/) how hard would it be to upgrade Maven to >= 3.3.1 on jenkins? [11:18:42] ideally 3.5... [11:19:17] stretch seems to provide 3.3.9... [11:34:12] hashar: I just realized that a similar job (https://gerrit.wikimedia.org/r/#/c/380327/) complete without error. Is it possible that we don't have the same version of maven on all slaves ? [11:34:51] I'm trying to dig into integration/config to see if we have a selector for the maven version, but so far, I'm failing to find my way... [11:54:24] !log Jenkins: removing the Violations plugin. It is not used. [11:54:29] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [11:54:57] gehel: hello. We dont do stretch yet, that depends on setting up docker on the infra which is a work in progress [11:55:14] gehel: so the images use Jessie for now, and maybe there is a more recent maven in jessie-backports? [11:55:34] 3.3.9-3 ! [11:55:56] Strange, one of the jobs completed... [11:56:09] and in puppet.git we install: manifests/packages/java.pp: package { 'maven2': [11:56:23] BUT [11:56:31] maybe the Jenkins job magically install maven for us [11:57:12] gehel: there is some tool setup in Jenkins at https://integration.wikimedia.org/ci/configureTools/ [11:57:24] Maven 2 is definitely a problem... [11:58:01] Need to run an errand, I'll have a look when back... [11:58:51] !log Jenkins: installed Warnings plugin [11:58:55] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [12:23:08] PROBLEM - Long lived cherry-picks on puppetmaster on deployment-puppetmaster02 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [12:35:48] PROBLEM - Mediawiki Error Rate on graphite-labs is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [10.0] [12:45:49] RECOVERY - Mediawiki Error Rate on graphite-labs is OK: OK: Less than 1.00% above the threshold [1.0] [12:59:15] hashar: I don't have access to that jenkins page you mentioned earlier... [12:59:30] I'll downgrade my Maven requirements for now, let's see if it breaks... [13:00:51] hashar: do you know the minimal version of maven that will be available on all slaves? [13:11:11] gehel: pretty sure that is maven2 [13:11:33] I'm surprised our build even work with maven 2! [13:11:43] but apparently Jenkins has support to add bunch of maven versions and we can have the job then point to whatever ver ( https://docs.openstack.org/infra/jenkins-job-builder/project_maven.html ) [13:11:47] so far, I haven't seen anything older than 3.0.1... [13:12:31] then I guess I have no clue :D [13:12:36] which job is it ? [13:13:11] https://integration.wikimedia.org/ci/job/search-highlighter and https://integration.wikimedia.org/ci/job/search-extra at the moment, but probably more to come... [13:14:54] ahhh [13:15:21] so yeah https://integration.wikimedia.org/ci/job/search-highlighter/configure [13:15:33] it seems to use Maven-3.0.4 [13:16:03] though maybe you no more have access to that page [13:16:28] nope, no access either, though this is probably something in jjb? [13:16:37] I guess the maven jobs pick whatever first maven is defined [13:16:41] could not find it either, but I'm not really sure what I am looking for... [13:16:49] since in jjb the jobs define no maven-name: [13:20:44] so by defining a maven-name, I could probably select the maven version I need... [13:22:31] !log Jenkins: adding Maven-3.0.5 to the tool configuration https://integration.wikimedia.org/ci/configureTools/ [13:22:35] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [13:23:14] gehel: it is reubilding at https://integration.wikimedia.org/ci/job/search-highlighter/566/console [13:23:23] will have to craft the relevant jjb patch though [13:23:47] hashar: no hurry, but it would be real nice to be able to ensure a specific maven versin! [13:24:56] (03PS1) 10Hashar: Bump some Maven jobs to use Maven 3.5.0 [integration/config] - 10https://gerrit.wikimedia.org/r/382696 [13:25:03] https://gerrit.wikimedia.org/r/382696 Bump some Maven jobs to use Maven 3.5.0 [13:25:05] that might do it [13:25:29] If that's all it take, great! [13:25:34] ;] [13:25:41] I only hacked the search highlighter one though [13:25:44] (03CR) 10Gehel: [C: 031] "LGTM, but I don't know much about JJB" [integration/config] - 10https://gerrit.wikimedia.org/r/382696 (owner: 10Hashar) [13:28:23] i gotta refresh the jobs and recheck all repos. But I first have to complete something else :] [13:34:44] hashar: no emergency! And thanks for the help! [14:27:11] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban), 10Operations, 10Jenkins: Upgrade jenkins to 2.73.1 (new lts release) - https://phabricator.wikimedia.org/T168644#3664532 (10MoritzMuehlenhoff) apt.wikimedia.org has been updated to 2.73.1. Let me know if I can help with anything... [14:43:22] (03CR) 10Hashar: [C: 032] "I have rebuild the jobs:" [integration/config] - 10https://gerrit.wikimedia.org/r/382696 (owner: 10Hashar) [14:44:35] (03Merged) 10jenkins-bot: Bump some Maven jobs to use Maven 3.5.0 [integration/config] - 10https://gerrit.wikimedia.org/r/382696 (owner: 10Hashar) [14:50:22] 10Gerrit, 10Release-Engineering-Team (Backlog), 10Operations, 10Patch-For-Review: Gerrit shows HTTP 500 error when pasting extended unicode characters - https://phabricator.wikimedia.org/T145885#3664629 (10Dzahn) [14:55:24] (03CR) 10Zoranzoki21: "Thank you very much!" [integration/config] - 10https://gerrit.wikimedia.org/r/380989 (owner: 10EddieGP) [15:04:20] (03PS1) 10Hashar: Job to run puppet-lint continuously [integration/config] - 10https://gerrit.wikimedia.org/r/382722 [15:07:19] 10Release-Engineering-Team (Kanban), 10Analytics-Kanban, 10Patch-For-Review: Move Wikistats 2 from Differential to Gerrit - https://phabricator.wikimedia.org/T177288#3664670 (10Nuria) [17:28:04] (03PS9) 10Thcipriani: docker: Recursively build images in order [integration/config] - 10https://gerrit.wikimedia.org/r/381761 (https://phabricator.wikimedia.org/T176318) (owner: 10Hashar) [17:28:06] (03PS1) 10Thcipriani: Dockerfiles: Add build.py tests [integration/config] - 10https://gerrit.wikimedia.org/r/382745 [17:29:08] (03CR) 10jerkins-bot: [V: 04-1] Dockerfiles: Add build.py tests [integration/config] - 10https://gerrit.wikimedia.org/r/382745 (owner: 10Thcipriani) [17:46:55] 10Release-Engineering-Team (Kanban), 10Patch-For-Review, 10Release, 10Train Deployments: 1.31.0-wmf.2 deployment blockers - https://phabricator.wikimedia.org/T174358#3665362 (10Jdforrester-WMF) [17:58:01] (03CR) 10Thcipriani: [C: 04-1] "I added some tests in I7d4df1443b9d50df45f7284af43e7cc1dee1dbca (which is currently failing tests due to: python 3.5 not found)." (032 comments) [integration/config] - 10https://gerrit.wikimedia.org/r/381761 (https://phabricator.wikimedia.org/T176318) (owner: 10Hashar) [18:04:21] (03PS2) 10Thcipriani: Dockerfiles: Add build.py tests [integration/config] - 10https://gerrit.wikimedia.org/r/382745 [18:05:26] (03CR) 10jerkins-bot: [V: 04-1] Dockerfiles: Add build.py tests [integration/config] - 10https://gerrit.wikimedia.org/r/382745 (owner: 10Thcipriani) [18:06:03] (03CR) 10Thcipriani: [C: 032] ">" (032 comments) [integration/config] - 10https://gerrit.wikimedia.org/r/381761 (https://phabricator.wikimedia.org/T176318) (owner: 10Hashar) [18:07:42] (03Merged) 10jenkins-bot: docker: Recursively build images in order [integration/config] - 10https://gerrit.wikimedia.org/r/381761 (https://phabricator.wikimedia.org/T176318) (owner: 10Hashar) [18:28:10] 10Release-Engineering-Team (Kanban), 10Release, 10Train Deployments: 1.31.0-wmf.3 deployment blockers - https://phabricator.wikimedia.org/T174359#3665533 (10greg) [19:16:44] Yippee, build fixed! [19:16:45] Project selenium-MinervaNeue ยป firefox,beta,Linux,BrowserTests build #152: 09FIXED in 27 min: https://integration.wikimedia.org/ci/job/selenium-MinervaNeue/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=BrowserTests/152/ [19:23:07] PROBLEM - SSH on integration-slave-docker-1001 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:56:32] hmm docker went down [19:56:36] [20:23:07] PROBLEM - SSH on integration-slave-docker-1001 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:57:24] hashar thcipriani ^^ [19:57:29] as operations ci uses that [19:57:37] dzahn got an aborted on his test [19:57:42] wonder if that's related? [19:57:47] https://integration.wikimedia.org/ci/job/operations-puppet-tests-docker/6821/console [19:57:56] (03CR) 10Hashar: "Looks good. Thanks for the tests!!" (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/382745 (owner: 10Thcipriani) [20:00:03] seems to work now. [20:11:00] paladox: that is the network flapping for some reason [20:11:05] paladox: "socket timeout" [20:11:13] ok thanks [20:21:44] 10Gerrit, 10MediaWiki-Vagrant, 10Patch-For-Review: "index-pack failed" when installing new MediaWiki-Vagrant box - https://phabricator.wikimedia.org/T152801#3666060 (10Kamsuri5) Thanks @Tgr, it's resolved. [20:43:50] (03PS1) 10Legoktm: doc: Add link for Equivset [integration/docroot] - 10https://gerrit.wikimedia.org/r/382806 [20:49:02] (03PS3) 10Thcipriani: Dockerfiles: Add build.py tests [integration/config] - 10https://gerrit.wikimedia.org/r/382745 [20:54:24] (03CR) 10Legoktm: [C: 032] doc: Add link for Equivset [integration/docroot] - 10https://gerrit.wikimedia.org/r/382806 (owner: 10Legoktm) [20:54:56] (03Merged) 10jenkins-bot: doc: Add link for Equivset [integration/docroot] - 10https://gerrit.wikimedia.org/r/382806 (owner: 10Legoktm) [20:55:02] (03CR) 10jenkins-bot: doc: Add link for Equivset [integration/docroot] - 10https://gerrit.wikimedia.org/r/382806 (owner: 10Legoktm) [20:56:14] (03PS4) 10Thcipriani: Dockerfiles: Add build.py tests [integration/config] - 10https://gerrit.wikimedia.org/r/382745 [20:56:22] thcipriani: tox is killing me :( [20:56:28] [testenv:dockerfiles] [20:56:28] changedir = dockerfiles [20:56:29] commands = /usr/bin/env tox [20:56:41] not working for you? [20:57:00] well it creates a /.tox/dockerfiles venv [20:57:11] and inherits deps from the [testenv] section [20:57:19] so that reinstall all the /test-requirements.txt dependnecies [20:57:26] ah blerg [20:57:46] I was just braindumping random weird thoughts :( [20:58:18] the one with toxworkdir = should work [20:59:06] or we move the deps out of [testenv] [21:00:39] hrm, the one with toxworkdir seems to share that behavior [21:00:46] yeah [21:01:44] thcipriani: how could we name the current tests in /tests/ ? Ithought of 'zuulworkflow' [21:01:54] or maybe just zuul hehe [21:02:36] yeah, zuul_tests [21:05:21] (03PS1) 10Hashar: Explicitly name the venv that run the zuul layout tests [integration/config] - 10https://gerrit.wikimedia.org/r/382835 [21:06:28] (03PS1) 10Hashar: tox: zuul is explicitly python2.7 [integration/config] - 10https://gerrit.wikimedia.org/r/382836 [21:06:51] (03Abandoned) 10Hashar: tox: zuul is explicitly python2.7 [integration/config] - 10https://gerrit.wikimedia.org/r/382836 (owner: 10Hashar) [21:06:56] bah [21:07:27] thcipriani: https://gerrit.wikimedia.org/r/382835 drops [testenv] that has bunch of default [21:09:04] (03PS5) 10Hashar: Dockerfiles: Add build.py tests [integration/config] - 10https://gerrit.wikimedia.org/r/382745 (owner: 10Thcipriani) [21:09:26] (03CR) 10Hashar: [C: 031] "Rebased on top of https://gerrit.wikimedia.org/r/#/c/382835/ :]" (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/382745 (owner: 10Thcipriani) [21:09:31] test are hard :) [21:09:36] looks good to me [21:09:41] feel free to CR+2 both of them :] [21:10:03] if I feel adventurous on Monday, I will probably rewrite the dockerfiles dependency from scratch [21:10:21] oh boy [21:10:23] I gotta find my reference book about graph theory [21:10:32] :P [21:10:38] really [21:10:44] if I had to hashar.reset() [21:10:57] I would go for graph theory :] [21:11:44] graphs are neat [21:12:07] https://networkx.github.io/ is veryyyyy nice [21:12:16] but it is probably too heavy for the issue we have [21:15:01] mayhaps [21:15:06] (03PS1) 10Legoktm: Set file filter for apps-android-wikipedia-tox-jessie [integration/config] - 10https://gerrit.wikimedia.org/r/382837 (https://phabricator.wikimedia.org/T177016) [21:15:28] is the current code not sufficient? [21:15:42] (03CR) 10Thcipriani: [C: 032] Explicitly name the venv that run the zuul layout tests [integration/config] - 10https://gerrit.wikimedia.org/r/382835 (owner: 10Hashar) [21:16:42] (03Merged) 10jenkins-bot: Explicitly name the venv that run the zuul layout tests [integration/config] - 10https://gerrit.wikimedia.org/r/382835 (owner: 10Hashar) [21:17:02] (03CR) 10Thcipriani: [C: 032] Dockerfiles: Add build.py tests [integration/config] - 10https://gerrit.wikimedia.org/r/382745 (owner: 10Thcipriani) [21:18:00] !! [21:18:09] (03Merged) 10jenkins-bot: Dockerfiles: Add build.py tests [integration/config] - 10https://gerrit.wikimedia.org/r/382745 (owner: 10Thcipriani) [21:18:12] you said feel free [21:19:04] and I did. [21:19:06] which would be the formal form for french "vas y merge les" ;D [21:19:30] (fr->en "common merge them") [21:19:51] anyway [21:20:06] gotta figure out a way to build those images on the CI infra [21:20:08] current code seems to work fine to me. I flailed at it a lot and it handled my flailing. [21:20:34] and by "flailed at it" I mean "wrote tests to verify my assumptions" [21:21:09] well now that docker is up-to-date on the ci infra it should be less-hard to do that. [21:21:28] d397e275c51e: Downloading [=> ] 729.7kB/24.09MB [21:21:28] 426a8c0d3c9f: Download complete [21:21:29] 80c07a0b1ed4: Downloading [=> ] 689.4kB/33.93MB [21:21:34] FUCK YOU DOCKER [21:21:55] huh [21:22:16] latest: Pulling from wmfreleng/ci-stretch [21:22:18] .... [21:25:16] we still need to figure out the tag system [21:25:29] whether we can just have a "stable" tag [21:27:14] hrm, seems to just shift around the problems from "latest" [21:27:38] or latest or whatever it's called - just not the date based tags [21:27:58] but on the other hand maybe all the problems of using the "latest" tag are in my head rather than practical considerations. [21:28:24] did we establish a way to make sure docker run will always use the latest latest? [21:28:50] IIRC you found an upstream task about adding a flag to docker run [21:29:39] something about a --pull=[esoteric-critera] flag [21:29:41] maybe the job that use docker run would have to docker pull first [21:29:42] https://github.com/moby/moby/issues/13331#issuecomment-104005676 [21:29:45] just to be sure [21:29:50] based on that we just need to run docker pull $image [21:30:00] addshore also mentionned using the commit sha1 as a tag [21:30:37] (03PS1) 10Legoktm: Move git cloning out of composer-package image [integration/config] - 10https://gerrit.wikimedia.org/r/382841 [21:30:50] sha1 does seem a bit more sane than timestamp after having used timestamp for a bit [21:31:04] hashar: or Gerrit change number [21:31:11] More readable than hash [21:31:19] 279345/4 or whatever [21:31:26] or we use git tags && git describe *evilness* [21:31:27] Or 279345 [21:31:48] the time to use git pull $image is only like a second [21:32:26] but that does make CI more dependent on docker hub than it is currently [21:32:34] (03CR) 10Hashar: "It is meant to run composer from integration/composer "easily". So that devs can locally:" [integration/config] - 10https://gerrit.wikimedia.org/r/382841 (owner: 10Legoktm) [21:33:06] what i thought of is we craft our patch +2 it, CI build the images and push them as :latest [21:33:11] the CI jobs all points to a :stable [21:33:31] and we manually (or via a job) tag the image :stable to promote it [21:33:41] but most probably I am overthinking the whole thing [21:34:14] or :latest everything and if something screw up: revert, rebuild, docker push [21:34:18] 10Continuous-Integration-Config, 10MediaWiki-Core-Tests, 10MediaWiki-Debug-Logger: Maintain debug settings in MediaWiki core - https://phabricator.wikimedia.org/T177669#3666254 (10Krinkle) [21:34:33] 10Continuous-Integration-Config, 10MediaWiki-Core-Tests, 10MediaWiki-Debug-Logger: Maintain debug settings in MediaWiki core - https://phabricator.wikimedia.org/T177669#3666266 (10Krinkle) [21:35:04] that seems pretty alright to me [21:35:20] (03CR) 10Legoktm: "Err, that seems a bit unrelated? This is the "composer-package" image. Mostly I'm doing this as a test to make sure this concept works, an" [integration/config] - 10https://gerrit.wikimedia.org/r/382841 (owner: 10Legoktm) [21:35:22] 10Continuous-Integration-Config, 10MediaWiki-Core-Tests, 10MediaWiki-Debug-Logger: Maintain debug settings in MediaWiki core - https://phabricator.wikimedia.org/T177669#3666272 (10Krinkle) [21:37:08] hashar: Is there a regular audit of beta cluster puppet cherry-picks? [21:37:27] Krinkle: not I know of [21:37:30] Would be good to clean them up at some point, and/or ensure follow up, or remove if not critical aafter a certain time. [21:37:58] Makes it harder to test other things. There's a fair number of apache config changes that appear to be meant as a test, but don't seem to have any current plans to move to prod. [21:39:00] right, where'd that task go... [21:39:28] https://phabricator.wikimedia.org/T135427 [21:39:31] that un ^ [21:40:38] moved to the back-burner but may still be a viable idea: https://gerrit.wikimedia.org/r/#/c/310719/ [21:40:53] thcipriani: hello. What did you end up doing to memc07? Last I know you were re-imaging. [21:41:36] AaronSchulz: yeah, I reimaged it. Should be up now, last I checked the firewall looked good on that one. Where "good" == "same as memc06". [21:42:55] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team (Backlog), 10Patch-For-Review: Beta puppetmaster cherry-pick process - https://phabricator.wikimedia.org/T135427#2298408 (10greg) was 23 in krenair's post above, now 18: ``` gjg@deployment-puppetmaster02:/var/lib/git/operations/puppet$ git log --one... [21:42:57] ok [21:45:41] 10Release-Engineering-Team, 10MediaWiki-extensions-WikibaseRepository, 10Wikidata, 10Beta-Cluster-reproducible, and 2 others: Jobs invoking SiteConfiguration::getConfig cause HHVM to fail updating the bytecode cache due to being filesize limited to 512MByte... - https://phabricator.wikimedia.org/T145819#3666291 [22:36:43] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team (Next), 10Patch-For-Review: Beta puppetmaster cherry-pick process - https://phabricator.wikimedia.org/T135427#3666393 (10greg) [22:37:10] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Backlog): Figure out the package conflict between libtidy-dev from sury and hhvm-tidy - https://phabricator.wikimedia.org/T169008#3666394 (10greg) [23:14:05] 10Beta-Cluster-Infrastructure, 10Multimedia, 10Structured-Data-Commons, 10Wikidata, and 4 others: Release FileAnnotations on the Beta Cluster - https://phabricator.wikimedia.org/T144302#3666472 (10Ramsey-WMF) Stalled because of wait for Wikibase on Commons, which is part of Structured Data on Commons [23:14:36] 10Beta-Cluster-Infrastructure, 10Structured-Data-Commons, 10Wikimedia-Extension-setup, 10Category, and 2 others: Release FileAnnotations on the Beta Cluster - https://phabricator.wikimedia.org/T144302#3666479 (10Ramsey-WMF) [23:43:44] (03PS2) 10Legoktm: Move git cloning out of composer-package image [integration/config] - 10https://gerrit.wikimedia.org/r/382841