[00:12:02] thcipriani ohhh another gerrit 2.16 feature is you can reload some configs from gerrit.config without restarting gerrit [00:12:03] thcipriani ohhh another gerrit 2.16 feature is you can reload some configs from gerrit.config without restarting gerrit [00:12:11] (not all of it's configs support that but some do) [00:12:11] (not all of it's configs support that but some do) [00:15:09] https://gerrit-documentation.storage.googleapis.com/Documentation/2.16/rest-api-config.html#reload-config [00:15:09] https://gerrit-documentation.storage.googleapis.com/Documentation/2.16/rest-api-config.html#reload-config [00:18:20] PROBLEM - SSH on deployment-rd3-cptest-master01 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:18:20] PROBLEM - SSH on deployment-rd3-cptest-master01 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:18:20] PROBLEM - SSH on deployment-imagescaler02 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:18:21] PROBLEM - SSH on deployment-imagescaler02 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:18:39] dosen't seem most of the configs support it [00:18:39] dosen't seem most of the configs support it [00:18:42] but at least 3 do [00:18:43] but at least 3 do [02:22:27] Project beta-scap-eqiad build #227865: 04FAILURE in 11 min: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/227865/ [02:22:28] Project beta-scap-eqiad build #227865: 04FAILURE in 11 min: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/227865/ [02:22:40] Project beta-update-databases-eqiad build #29808: 04FAILURE in 2 min 40 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/29808/ [02:22:40] Project beta-update-databases-eqiad build #29808: 04FAILURE in 2 min 40 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/29808/ [02:37:02] Yippee, build fixed! [02:37:02] Yippee, build fixed! [02:37:02] Project beta-scap-eqiad build #227866: 09FIXED in 12 min: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/227866/ [02:37:03] Project beta-scap-eqiad build #227866: 09FIXED in 12 min: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/227866/ [03:22:19] Yippee, build fixed! [03:22:19] Yippee, build fixed! [03:22:20] Project beta-update-databases-eqiad build #29809: 09FIXED in 2 min 19 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/29809/ [03:22:20] Project beta-update-databases-eqiad build #29809: 09FIXED in 2 min 19 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/29809/ [04:11:48] PROBLEM - Host integration-slave-docker-1041 is DOWN: CRITICAL - Host Unreachable (10.68.19.23) [04:11:48] PROBLEM - Host integration-slave-docker-1041 is DOWN: CRITICAL - Host Unreachable (10.68.19.23) [08:57:27] 10Continuous-Integration-Infrastructure (shipyard), 10Release-Engineering-Team (Kanban), 10Operations, 10Patch-For-Review, 10cloud-services-team (Kanban): Phase out Nodepool from production - https://phabricator.wikimedia.org/T209361 (10aborrero) [08:57:27] 10Continuous-Integration-Infrastructure (shipyard), 10Release-Engineering-Team (Kanban), 10Operations, 10Patch-For-Review, 10cloud-services-team (Kanban): Phase out Nodepool from production - https://phabricator.wikimedia.org/T209361 (10aborrero) [09:00:32] 10Continuous-Integration-Infrastructure (shipyard), 10Release-Engineering-Team (Kanban), 10Operations, 10Patch-For-Review, 10cloud-services-team (Kanban): Remove labnodepool1001.eqiad.wmnet - https://phabricator.wikimedia.org/T209642 (10aborrero) >>! In T209642#4752043, @hashar wrote: > #cloud-services-t... [09:00:32] 10Continuous-Integration-Infrastructure (shipyard), 10Release-Engineering-Team (Kanban), 10Operations, 10Patch-For-Review, 10cloud-services-team (Kanban): Remove labnodepool1001.eqiad.wmnet - https://phabricator.wikimedia.org/T209642 (10aborrero) >>! In T209642#4752043, @hashar wrote: > #cloud-services-t... [09:06:54] 10Continuous-Integration-Infrastructure (shipyard), 10Release-Engineering-Team (Kanban), 10Operations, 10decommission, and 2 others: Remove labnodepool1001.eqiad.wmnet - https://phabricator.wikimedia.org/T209642 (10MoritzMuehlenhoff) [09:06:55] 10Continuous-Integration-Infrastructure (shipyard), 10Release-Engineering-Team (Kanban), 10Operations, 10decommission, and 2 others: Remove labnodepool1001.eqiad.wmnet - https://phabricator.wikimedia.org/T209642 (10MoritzMuehlenhoff) [09:26:40] (03PS1) 10WMDE-Fisch: Add quibble job for selenium tests on Cite [integration/config] - 10https://gerrit.wikimedia.org/r/474134 (https://phabricator.wikimedia.org/T205270) [09:26:40] (03PS1) 10WMDE-Fisch: Add quibble job for selenium tests on Cite [integration/config] - 10https://gerrit.wikimedia.org/r/474134 (https://phabricator.wikimedia.org/T205270) [09:29:51] (03CR) 10jerkins-bot: [V: 04-1] Add quibble job for selenium tests on Cite [integration/config] - 10https://gerrit.wikimedia.org/r/474134 (https://phabricator.wikimedia.org/T205270) (owner: 10WMDE-Fisch) [09:29:51] (03CR) 10jerkins-bot: [V: 04-1] Add quibble job for selenium tests on Cite [integration/config] - 10https://gerrit.wikimedia.org/r/474134 (https://phabricator.wikimedia.org/T205270) (owner: 10WMDE-Fisch) [09:38:04] (03CR) 10Zfilipin: "PS8 is tested with https://gerrit.wikimedia.org/r/c/search/highlighter/+/473766" [integration/config] - 10https://gerrit.wikimedia.org/r/472661 (https://phabricator.wikimedia.org/T207046) (owner: 10Gehel) [09:38:04] (03CR) 10Zfilipin: "PS8 is tested with https://gerrit.wikimedia.org/r/c/search/highlighter/+/473766" [integration/config] - 10https://gerrit.wikimedia.org/r/472661 (https://phabricator.wikimedia.org/T207046) (owner: 10Gehel) [09:39:50] (03CR) 10Zfilipin: "The job failed" [integration/config] - 10https://gerrit.wikimedia.org/r/472661 (https://phabricator.wikimedia.org/T207046) (owner: 10Gehel) [09:39:50] (03CR) 10Zfilipin: "The job failed" [integration/config] - 10https://gerrit.wikimedia.org/r/472661 (https://phabricator.wikimedia.org/T207046) (owner: 10Gehel) [09:39:51] gehel: bonjour :) Zeljko poked me about adding sonar to some jobs https://gerrit.wikimedia.org/r/c/integration/config/+/472661 [09:39:51] gehel: bonjour :) Zeljko poked me about adding sonar to some jobs https://gerrit.wikimedia.org/r/c/integration/config/+/472661 [09:39:59] I have lost track of the effort oon adding SonarQube [09:39:59] I have lost track of the effort oon adding SonarQube [09:40:23] (03PS2) 10WMDE-Fisch: Add experimental selenium tests on Cite [integration/config] - 10https://gerrit.wikimedia.org/r/474134 (https://phabricator.wikimedia.org/T205270) [09:40:23] (03PS2) 10WMDE-Fisch: Add experimental selenium tests on Cite [integration/config] - 10https://gerrit.wikimedia.org/r/474134 (https://phabricator.wikimedia.org/T205270) [09:40:33] and I don't think we can use the Jenkins SonarQube plugin, it is probably not going to be compatible with how we run the maven jobs: in docker containers [09:40:33] and I don't think we can use the Jenkins SonarQube plugin, it is probably not going to be compatible with how we run the maven jobs: in docker containers [09:42:21] (03CR) 10jerkins-bot: [V: 04-1] Add experimental selenium tests on Cite [integration/config] - 10https://gerrit.wikimedia.org/r/474134 (https://phabricator.wikimedia.org/T205270) (owner: 10WMDE-Fisch) [09:42:21] (03CR) 10jerkins-bot: [V: 04-1] Add experimental selenium tests on Cite [integration/config] - 10https://gerrit.wikimedia.org/r/474134 (https://phabricator.wikimedia.org/T205270) (owner: 10WMDE-Fisch) [09:42:58] hashar: yeah, we don't really need that plugin at the moment (we'll need something at some point, but we can get back to that later) [09:42:58] hashar: yeah, we don't really need that plugin at the moment (we'll need something at some point, but we can get back to that later) [09:43:29] gehel: zeljkof: and I was obsessive on setting up our own SonarQube server, but it is probably better to just use sonarcloud for now isn't it ? [09:43:29] gehel: zeljkof: and I was obsessive on setting up our own SonarQube server, but it is probably better to just use sonarcloud for now isn't it ? [09:43:38] and maybe later on add our own instance [09:43:38] and maybe later on add our own instance [09:43:40] we're experimenting with zeljkof, but we're having issues: https://integration.wikimedia.org/ci/job/search-highlighter-maven-java8-docker-site-publish/22/console [09:43:40] we're experimenting with zeljkof, but we're having issues: https://integration.wikimedia.org/ci/job/search-highlighter-maven-java8-docker-site-publish/22/console [09:44:09] hashar: yes, we think sonar cloud is good for now [09:44:09] hashar: yes, we think sonar cloud is good for now [09:44:22] yeah, we're experimenting at the moment, making sure it will actually work. Better do that testing with sonarcloud, and if it is good enough for us, we'll install it in house [09:44:22] yeah, we're experimenting at the moment, making sure it will actually work. Better do that testing with sonarcloud, and if it is good enough for us, we'll install it in house [09:44:41] right now, we're trying to run a local scan, then send data to sonar cloud [09:44:41] right now, we're trying to run a local scan, then send data to sonar cloud [09:44:46] does not make sense to waste time on it before we're confident it is the right solution [09:44:46] does not make sense to waste time on it before we're confident it is the right solution [09:45:20] the failing job (link above) has issues creating a dir: [09:45:20] the failing job (link above) has issues creating a dir: [09:45:21] Unable to create user cache: /srv/jenkins-workspace/workspace/search-highlighter-maven-java8-docker-site-publish/.sonar/cache [09:45:21] Unable to create user cache: /srv/jenkins-workspace/workspace/search-highlighter-maven-java8-docker-site-publish/.sonar/cache [09:45:28] and things should work fine for java projects, getting sonar from maven, but we can't figure out how to give the job a local folder to write the data to [09:45:28] and things should work fine for java projects, getting sonar from maven, but we can't figure out how to give the job a local folder to write the data to [09:45:43] this what we have so far [09:45:43] this what we have so far [09:45:48] `options: '--entrypoint=/usr/local/bin/mvn --env SONAR_USER_HOME="$WORKSPACE"/.sonar'` [09:45:48] `options: '--entrypoint=/usr/local/bin/mvn --env SONAR_USER_HOME="$WORKSPACE"/.sonar'` [09:46:08] SONAR_USER_HOME is what needs to be writable [09:46:08] SONAR_USER_HOME is what needs to be writable [09:46:33] well, it needs to point to a folder writable by the job/container, however it works now [09:46:33] well, it needs to point to a folder writable by the job/container, however it works now [09:46:52] Tools are expected to honor the env variable XDG_CACHE_HOME [09:46:52] Tools are expected to honor the env variable XDG_CACHE_HOME [09:46:58] the containers have ENV XDG_CACHE_HOME=/cache set [09:46:58] the containers have ENV XDG_CACHE_HOME=/cache set [09:47:02] should it be something like `SONAR_USER_HOME="$WORKSPACE"/.sonar:sonar` [09:47:02] should it be something like `SONAR_USER_HOME="$WORKSPACE"/.sonar:sonar` [09:47:18] ah, we can use that folder? [09:47:18] ah, we can use that folder? [09:47:25] the CI job restore the cache from a central place (castor) to $WORKSPACE/cache which is then mounted inside the counter as /cache [09:47:25] the CI job restore the cache from a central place (castor) to $WORKSPACE/cache which is then mounted inside the counter as /cache [09:47:43] and on a successfull gate and submit, that cache directory is saved to the central cache [09:47:43] and on a successfull gate and submit, that cache directory is saved to the central cache [09:47:57] seems Sonar does not honor XDG_CACHE_HOME :((( [09:47:57] seems Sonar does not honor XDG_CACHE_HOME :((( [09:48:30] does not seem like it does! [09:48:31] does not seem like it does! [09:48:54] the job failed when SONAR_USER_HOME is not set [09:48:54] the job failed when SONAR_USER_HOME is not set [09:48:56] see https://integration.wikimedia.org/ci/job/search-extra-analysis-maven-java8-docker-site-publish/26/console [09:48:56] see https://integration.wikimedia.org/ci/job/search-extra-analysis-maven-java8-docker-site-publish/26/console [09:49:15] Failed to execute goal org.sonarsource.scanner.maven:sonar-maven-plugin:3.5.0.1254:sonar (default-cli) on project extra-analysis: Unable to create user cache: /nonexistent/.sonar/cache -> [Help 1] [09:49:15] Failed to execute goal org.sonarsource.scanner.maven:sonar-maven-plugin:3.5.0.1254:sonar (default-cli) on project extra-analysis: Unable to create user cache: /nonexistent/.sonar/cache -> [Help 1] [09:49:18] ok, so we should set SONAR_CACHE_HOME=${XDG_CACHE_HOME}/.sonar [09:49:18] ok, so we should set SONAR_CACHE_HOME=${XDG_CACHE_HOME}/.sonar [09:49:31] also when you pass to docker run : --env SONAR_USER_HOME="$WORKSPACE"/.sonar , $WORKSPACE is coming from the jenkins slave host and is evaluated in that context, so you end up with something like /srv/jenkins-workspace/maven-docker/.sonar which obviously does NOT exist inside the container [09:49:31] also when you pass to docker run : --env SONAR_USER_HOME="$WORKSPACE"/.sonar , $WORKSPACE is coming from the jenkins slave host and is evaluated in that context, so you end up with something like /srv/jenkins-workspace/maven-docker/.sonar which obviously does NOT exist inside the container [09:49:50] ditto SONAR_CACHE_HOME=${XDG_CACHE_HOME}/.sonar the env variable is evaluated on the host [09:49:50] ditto SONAR_CACHE_HOME=${XDG_CACHE_HOME}/.sonar the env variable is evaluated on the host [09:49:55] not obviously for me :P [09:49:55] not obviously for me :P [09:49:55] ofc [09:49:56] ofc [09:50:08] my docker-fu is really bad [09:50:08] my docker-fu is really bad [09:51:09] hashar: so, what the value of SONAR_USER_HOME should be? [09:51:09] hashar: so, what the value of SONAR_USER_HOME should be? [09:51:11] * zeljkof is lost [09:51:11] ACTION is lost [09:51:23] sonar is run inside the container [09:51:23] sonar is run inside the container [09:51:46] CI runs the container with $WORKSPACE/cache directory from the jenkins slave being mounted inside the container as /cache / [09:51:46] CI runs the container with $WORKSPACE/cache directory from the jenkins slave being mounted inside the container as /cache / [09:51:53] so /cache/.sonar? [09:51:53] so /cache/.sonar? [09:52:02] ( docker run -v "$(pwd)/.cache":/cache ) [09:52:02] ( docker run -v "$(pwd)/.cache":/cache ) [09:52:38] --env SONAR_USER_HOME=/cache/.sonar [09:52:38] --env SONAR_USER_HOME=/cache/.sonar [09:52:45] hashar: like that? ^ [09:52:46] hashar: like that? ^ [09:52:56] so for maven we went with: [09:52:56] so for maven we went with: [09:53:09] $ cat dockerfiles/java8/settings.xml [09:53:09] $ cat dockerfiles/java8/settings.xml [09:53:09] [09:53:09] ${env.XDG_CACHE_HOME}/maven [09:53:09] [09:53:09] [09:53:09] ${env.XDG_CACHE_HOME}/maven [09:53:09] [09:53:41] we could add that to settings as well [09:53:42] we could add that to settings as well [09:53:46] lemme check how [09:53:46] lemme check how [09:53:57] that settings.xml is copied in the image as /settings.xml and we then invoke maven using: mvn -gs /settings.xml [09:53:57] that settings.xml is copied in the image as /settings.xml and we then invoke maven using: mvn -gs /settings.xml [09:54:19] gehel: that would be ideal yeah [09:54:19] gehel: that would be ideal yeah [09:54:46] then have that settings file to set SONAR_USER_HOME=${env.XDG_CACHE_HOME}/sonar [09:54:46] then have that settings file to set SONAR_USER_HOME=${env.XDG_CACHE_HOME}/sonar [09:54:51] so tldr: we don't need SONAR_USER_HOME, but we can set something in maven? [09:54:51] so tldr: we don't need SONAR_USER_HOME, but we can set something in maven? [09:55:08] apparently we do need it [09:55:08] apparently we do need it [09:55:24] but it is probably better to have the container to hold that configuration [09:55:24] but it is probably better to have the container to hold that configuration [09:55:39] (so that eg when you run the container on your local machine, the sonar cache is written at the right place) [09:55:39] (so that eg when you run the container on your local machine, the sonar cache is written at the right place) [09:55:58] else one running the container locally would have to know to pass --env SONAR_USER_HOME=/cache/sonar [09:55:58] else one running the container locally would have to know to pass --env SONAR_USER_HOME=/cache/sonar [09:57:07] and we could also set sonar.login, which would be nicer than what I'm currently doing in the pom of each project [09:57:07] and we could also set sonar.login, which would be nicer than what I'm currently doing in the pom of each project [09:57:22] you lost me at the first mention of docker, or maybe container ;P [09:57:22] you lost me at the first mention of docker, or maybe container ;P [09:58:00] ok, the param is not clearly documented, time to dive into the code [09:58:00] ok, the param is not clearly documented, time to dive into the code [09:58:13] send a search and rescue team if you don't hear from me in the next few hours [09:58:13] send a search and rescue team if you don't hear from me in the next few hours [09:58:16] git clone --depth=1 https://github.com/SonarSource/sonarqube ? :) [09:58:16] git clone --depth=1 https://github.com/SonarSource/sonarqube ? :) [09:58:32] :) [09:58:32] :) [09:58:38] well send rachel [09:58:38] well send rachel [09:58:50] :) [09:58:50] :) [09:59:00] hashar: nope, https://github.com/SonarSource/sonar-scanner-maven/ [09:59:00] hashar: nope, https://github.com/SonarSource/sonar-scanner-maven/ [09:59:01] that git repo does not have any match for "XDG" :( [09:59:01] that git repo does not have any match for "XDG" :( [09:59:33] that one does not either bah [09:59:33] that one does not either bah [10:00:07] I would have been surprised if it did [10:00:07] I would have been surprised if it did [10:00:12] * hashar tries https://github.com/search?q=org%3ASonarSource+SONAR_USER_HOME&unscoped_q=SONAR_USER_HOME [10:00:12] ACTION tries https://github.com/search?q=org%3ASonarSource+SONAR_USER_HOME&unscoped_q=SONAR_USER_HOME [10:02:13] ah there is sonar-scanner-api which has a api/src/main/java/org/sonarsource/scanner/api/internal/cache/FileCache.java [10:02:13] ah there is sonar-scanner-api which has a api/src/main/java/org/sonarsource/scanner/api/internal/cache/FileCache.java [10:03:06] yeah, I'm going to open a bug against sonar-scanner [10:03:06] yeah, I'm going to open a bug against sonar-scanner [10:03:10] https://github.com/SonarSource/sonar-scanner-api/ [10:03:10] https://github.com/SonarSource/sonar-scanner-api/ [10:03:20] api/src/main/java/org/sonarsource/scanner/api/internal/cache/FileCacheBuilder.java: String path = System.getenv("SONAR_USER_HOME"); [10:03:20] api/src/main/java/org/sonarsource/scanner/api/internal/cache/FileCacheBuilder.java: String path = System.getenv("SONAR_USER_HOME"); [10:03:26] but we should have a cleaner way to pass that the maven plugin [10:03:26] but we should have a cleaner way to pass that the maven plugin [10:03:48] I found that via github search for SONAR_USER_HOME accross the whole SonarSource organization: https://github.com/search?q=org%3ASonarSource+SONAR_USER_HOME&unscoped_q=SONAR_USER_HOME [10:03:48] I found that via github search for SONAR_USER_HOME accross the whole SonarSource organization: https://github.com/search?q=org%3ASonarSource+SONAR_USER_HOME&unscoped_q=SONAR_USER_HOME [10:04:38] nah, seems there isn't [10:04:38] nah, seems there isn't [10:04:44] https://github.com/SonarSource/sonar-scanner-api/blob/4ba28713442f1074d4283fd72cd25518eb6dbbc2/api/src/main/java/org/sonarsource/scanner/api/internal/cache/FileCacheBuilder.java#L43-L58 [10:04:44] https://github.com/SonarSource/sonar-scanner-api/blob/4ba28713442f1074d4283fd72cd25518eb6dbbc2/api/src/main/java/org/sonarsource/scanner/api/internal/cache/FileCacheBuilder.java#L43-L58 [10:04:54] so probably HAVE to use SONAR_USER_HOME [10:04:54] so probably HAVE to use SONAR_USER_HOME [10:04:59] so back to setting env vars from settings [10:04:59] so back to setting env vars from settings [10:05:09] if we could set that env variable in the maven settings.xml file which is in the java8 container, that would be ideal [10:05:09] if we could set that env variable in the maven settings.xml file which is in the java8 container, that would be ideal [10:05:23] not even sure if that's possible [10:05:23] not even sure if that's possible [10:05:27] and if you like to waste time, maybe fill an issue for them to support XDG directories [10:05:27] and if you like to waste time, maybe fill an issue for them to support XDG directories [10:05:35] alternative [10:05:35] alternative [10:05:59] is to add the env variable in the container ( ENV SONAR_USER_HOME=$XDG_CACHE_HOME/sonar ) might work [10:05:59] is to add the env variable in the container ( ENV SONAR_USER_HOME=$XDG_CACHE_HOME/sonar ) might work [10:07:26] doesnt look like we can set environment from settings.xml, only ready it [10:07:26] doesnt look like we can set environment from settings.xml, only ready it [10:08:15] but in that case, XDG_CACHE_HOME will be evaluated outside the container [10:08:15] but in that case, XDG_CACHE_HOME will be evaluated outside the container [10:11:02] should we hardcode SONAR_USER_HOME=/cache/.sonar ? [10:11:02] should we hardcode SONAR_USER_HOME=/cache/.sonar ? [10:11:17] that seems not much worse [10:11:17] that seems not much worse [10:12:19] ENV XDG_CACHE_HOME=/cache [10:12:19] ENV XDG_CACHE_HOME=/cache [10:12:31] ENV SONAR_USER_HOME=$XDG_CACHE_HOME/sonar [10:12:31] ENV SONAR_USER_HOME=$XDG_CACHE_HOME/sonar [10:12:34] I think that will work [10:12:34] I think that will work [10:16:03] yeah it does [10:16:03] yeah it does [10:17:33] FROM docker-registry.wikimedia.org/releng/ci-stretch [10:17:33] FROM docker-registry.wikimedia.org/releng/ci-stretch [10:17:34] ENV SONAR_USER_HOME=$XDG_CACHE_HOME/sonar [10:17:34] ENTRYPOINT /usr/bin/printf "Sonar user home is %s\n" $SONAR_USER_HOME [10:17:34] ENV SONAR_USER_HOME=$XDG_CACHE_HOME/sonar [10:17:34] ENTRYPOINT /usr/bin/printf "Sonar user home is %s\n" $SONAR_USER_HOME [10:17:46] kool! [10:17:46] kool! [10:17:46] then: [10:17:46] then: [10:17:49] docker build -t hashartest . && docker run --rm -it hashartest [10:17:49] docker build -t hashartest . && docker run --rm -it hashartest [10:17:59] Successfully tagged hashartest:latest [10:17:59] Successfully tagged hashartest:latest [10:17:59] Sonar user home is /cache/sonar [10:17:59] Sonar user home is /cache/sonar [10:18:20] so I guess, lets add that to the releng/java8 container. That is not so great, but it seems an acceptable workaround [10:18:20] so I guess, lets add that to the releng/java8 container. That is not so great, but it seems an acceptable workaround [10:19:32] so I guess edit dockerfiles/java8/Dockerfile.template , somewhere at the top add ENV SONAR_USER_HOME=$XDG_CACHE_HOME/sonar [10:19:33] so I guess edit dockerfiles/java8/Dockerfile.template , somewhere at the top add ENV SONAR_USER_HOME=$XDG_CACHE_HOME/sonar [10:19:36] * gehel needs coffee [10:19:36] ACTION needs coffee [10:19:45] dch -i -c dockerfiles/java8/changelog [10:19:45] dch -i -c dockerfiles/java8/changelog [10:19:47] and we are set :) [10:19:47] I finally found where to open tickets to sonar [10:19:47] and we are set :) [10:19:47] I finally found where to open tickets to sonar [10:19:52] ahah [10:19:52] ahah [10:19:53] jira? :) [10:19:54] jira? :) [10:20:03] nope, that would have been easy! [10:20:03] nope, that would have been easy! [10:20:22] is something up with contint? [10:20:22] is something up with contint? [10:20:27] I just got mv: cannot stat '/srv/workspace/puppet/.tox/log/*' [10:20:27] I just got mv: cannot stat '/srv/workspace/puppet/.tox/log/*' [10:20:27] they don't want us to mess with their jira, so there is a forum [10:20:27] they don't want us to mess with their jira, so there is a forum [10:20:33] https://integration.wikimedia.org/ci/job/operations-puppet-tests-stretch-docker/695/console [10:20:34] https://integration.wikimedia.org/ci/job/operations-puppet-tests-stretch-docker/695/console [10:21:56] hashar: any ideas? [10:21:56] hashar: any ideas? [10:27:06] apergos: that is probably just a warning [10:27:06] apergos: that is probably just a warning [10:27:54] apergos: if you have ruby / bundler / python / tox etc installed on your local machine, you should be able to reproduce with: bundle install && bundle exec rake --job=1 test [10:27:54] apergos: if you have ruby / bundler / python / tox etc installed on your local machine, you should be able to reproduce with: bundle install && bundle exec rake --job=1 test [10:28:06] that will run the tasks serially which makes it easier to find what failed [10:28:06] that will run the tasks serially which makes it easier to find what failed [10:28:28] on the CI job, it defaults to run the task sin a parallel hence outputs are multiplexed in the console output :/ [10:28:28] on the CI job, it defaults to run the task sin a parallel hence outputs are multiplexed in the console output :/ [10:28:31] 10:18:09 + git remote add zuul git://contint2001.wikimedia.org/operations/puppet 10:18:09 + git pull --quiet zuul production 10:18:09 fatal: read error: Connection reset by peer [10:28:31] 10:18:09 + git remote add zuul git://contint2001.wikimedia.org/operations/puppet 10:18:09 + git pull --quiet zuul production 10:18:09 fatal: read error: Connection reset by peer [10:28:34] how about these? [10:28:34] how about these? [10:28:40] ah [10:28:40] ah [10:28:54] yeah [10:28:54] yeah [10:29:08] apergos: the job failed to fetch the repo/patch from the CI git server contint2001 [10:29:08] apergos: the job failed to fetch the repo/patch from the CI git server contint2001 [10:29:14] maybe it is dead or it is a cosmic ray :( [10:29:14] maybe it is dead or it is a cosmic ray :( [10:29:16] that is what it looks like, yes [10:29:16] that is what it looks like, yes [10:29:21] do I just recheck and hope? [10:29:21] do I just recheck and hope? [10:29:33] yeah recheck [10:29:34] yeah recheck [10:29:36] k [10:29:36] k [10:29:41] the git-daemon is running on contint2001 at least [10:29:41] the git-daemon is running on contint2001 at least [10:30:28] god [10:30:28] god [10:30:29] *good [10:30:29] *good [10:30:35] maybe I was just unlucky, we'll see [10:30:35] maybe I was just unlucky, we'll see [10:31:09] yay my luck [10:31:09] yay my luck [10:31:12] Nov 16 10:18:09 contint2001 git-daemon[1131]: fatal: packet write with format failed: Connection reset by peer [10:31:13] Nov 16 10:18:09 contint2001 git-daemon[1131]: fatal: packet write with format failed: Connection reset by peer [10:31:14] (ok now) [10:31:14] (ok now) [10:31:15] that is all I have :( [10:31:15] that is all I have :( [10:31:17] (03Abandoned) 10WMDE-Fisch: Add experimental selenium tests on Cite [integration/config] - 10https://gerrit.wikimedia.org/r/474134 (https://phabricator.wikimedia.org/T205270) (owner: 10WMDE-Fisch) [10:31:17] (03Abandoned) 10WMDE-Fisch: Add experimental selenium tests on Cite [integration/config] - 10https://gerrit.wikimedia.org/r/474134 (https://phabricator.wikimedia.org/T205270) (owner: 10WMDE-Fisch) [10:31:18] huh great [10:31:18] huh great [10:31:28] hard to debug a transient error with that [10:31:28] hard to debug a transient error with that [10:35:26] 10MediaWiki-Codesniffer, 10goodfirstbug: Add sniff to detect usage of deprecated $wgUser - https://phabricator.wikimedia.org/T50963 (10Daimona) So, what's the status here? Adding wgUser [[https://github.com/wikimedia/mediawiki-tools-codesniffer/blob/a46f7fcc8b087c874a51e458bb34d5149a1d3546/MediaWiki/Sniffs/Var... [10:35:26] 10MediaWiki-Codesniffer, 10goodfirstbug: Add sniff to detect usage of deprecated $wgUser - https://phabricator.wikimedia.org/T50963 (10Daimona) So, what's the status here? Adding wgUser [[https://github.com/wikimedia/mediawiki-tools-codesniffer/blob/a46f7fcc8b087c874a51e458bb34d5149a1d3546/MediaWiki/Sniffs/Var... [10:42:02] hashar, zeljkof: https://community.sonarsource.com/t/sonar-scanner-should-honor-xdg-cache-home/4339 [10:42:02] hashar, zeljkof: https://community.sonarsource.com/t/sonar-scanner-should-honor-xdg-cache-home/4339 [10:43:36] 10Release-Engineering-Team (Kanban), 10Code-Health-Metrics, 10Patch-For-Review, 10User-zeljkofilipin: Code health metrics spike - https://phabricator.wikimedia.org/T207046 (10zeljkofilipin) @Gehel reported a Sonar bug https://community.sonarsource.com/t/sonar-scanner-should-honor-xdg-cache-home/4339 [10:43:36] 10Release-Engineering-Team (Kanban), 10Code-Health-Metrics, 10Patch-For-Review, 10User-zeljkofilipin: Code health metrics spike - https://phabricator.wikimedia.org/T207046 (10zeljkofilipin) @Gehel reported a Sonar bug https://community.sonarsource.com/t/sonar-scanner-should-honor-xdg-cache-home/4339 [10:43:45] gehel: added it to the task https://phabricator.wikimedia.org/T207046#4753282 [10:43:46] gehel: added it to the task https://phabricator.wikimedia.org/T207046#4753282 [10:43:53] zeljkof: thanks! [10:43:53] zeljkof: thanks! [10:44:00] * gehel is looking at that dockerfile [10:44:00] ACTION is looking at that dockerfile [10:44:03] gehel: can we work around it? [10:44:03] gehel: can we work around it? [10:44:17] yeah we should be able to [10:44:17] yeah we should be able to [10:44:23] cool [10:44:23] cool [10:44:30] do you need help? can you update the jjb? [10:44:30] do you need help? can you update the jjb? [10:44:31] * gehel does not know much about docker,, but sedning a patch in a few minutes [10:44:32] ACTION does not know much about docker,, but sedning a patch in a few minutes [10:44:39] great [10:44:39] great [10:44:55] gehel: thank you to have reported the issue to upstream [10:44:55] gehel: thank you to have reported the issue to upstream [10:45:17] not sure they'll do anything about it [10:45:17] not sure they'll do anything about it [10:45:56] it would be kind of surprising for most users to suddenly have their cache move from $HOME/.sonar/cache to somewhere else [10:45:56] it would be kind of surprising for most users to suddenly have their cache move from $HOME/.sonar/cache to somewhere else [10:46:49] well [10:46:49] well [10:47:08] based on xdg that would end up being in ~/.cache/sonar/ [10:47:08] based on xdg that would end up being in ~/.cache/sonar/ [10:47:19] and config files in ~/.config/sonar/ [10:47:19] and config files in ~/.config/sonar/ [10:47:52] I'm not saying it does not make sense, just that it would be somewhat of a breaking change [10:47:52] I'm not saying it does not make sense, just that it would be somewhat of a breaking change [10:47:56] and not portable [10:47:56] and not portable [10:49:22] https://wiki.archlinux.org/index.php/XDG_Base_Directory is somewhat a good read :) [10:49:22] https://wiki.archlinux.org/index.php/XDG_Base_Directory is somewhat a good read :) [10:49:33] ( archlinux wiki has a ton of useful docs about random topics) [10:49:34] ( archlinux wiki has a ton of useful docs about random topics) [10:50:12] maintenance-disconnect-full-disks build 20714 integration-slave-jessie-1003 (/srv: 95%): OFFLINE due to disk space [10:50:12] maintenance-disconnect-full-disks build 20714 integration-slave-jessie-1003 (/srv: 95%): OFFLINE due to disk space [10:53:48] (03PS1) 10Gehel: Add sonarcloud publication to maven site publish template [integration/config] - 10https://gerrit.wikimedia.org/r/474148 [10:53:48] (03PS1) 10Gehel: Add sonarcloud publication to maven site publish template [integration/config] - 10https://gerrit.wikimedia.org/r/474148 [10:54:08] hashar, zeljkof :https://gerrit.wikimedia.org/r/c/integration/config/+/474148 [10:54:08] hashar, zeljkof :https://gerrit.wikimedia.org/r/c/integration/config/+/474148 [10:54:35] ^ I'm partly guessing at how all this works, and not entirely sure how to test it [10:54:35] ^ I'm partly guessing at how all this works, and not entirely sure how to test it [10:55:02] but that might just work :) [10:55:02] but that might just work :) [10:55:12] maintenance-disconnect-full-disks build 20715 integration-slave-jessie-1003: OFFLINE due to disk space [10:55:12] maintenance-disconnect-full-disks build 20715 integration-slave-jessie-1003: OFFLINE due to disk space [10:56:08] gehel: why a separate commit? [10:56:08] gehel: why a separate commit? [10:56:31] why didn't you amend the previous commit? [10:56:31] why didn't you amend the previous commit? [10:56:56] to test it, I can deploy the jobs and you can merge another patch, like we did so far, right? [10:56:56] to test it, I can deploy the jobs and you can merge another patch, like we did so far, right? [10:57:37] separate commit because I wrote before thinking :( [10:57:37] separate commit because I wrote before thinking :( [10:57:59] lemme clean that up [10:57:59] lemme clean that up [10:58:18] and I guess, hashar needs to deploy the new docker container? [10:58:18] and I guess, hashar needs to deploy the new docker container? [10:58:33] probably, I have no idea how that part works [10:58:33] probably, I have no idea how that part works [10:58:36] if you changed it, I'm not sure how that's done [10:58:36] if you changed it, I'm not sure how that's done [10:59:54] (03PS9) 10Gehel: Add sonarcloud publication to maven site publish template [integration/config] - 10https://gerrit.wikimedia.org/r/472661 (https://phabricator.wikimedia.org/T207046) [10:59:54] (03PS9) 10Gehel: Add sonarcloud publication to maven site publish template [integration/config] - 10https://gerrit.wikimedia.org/r/472661 (https://phabricator.wikimedia.org/T207046) [11:00:07] (03Abandoned) 10Gehel: Add sonarcloud publication to maven site publish template [integration/config] - 10https://gerrit.wikimedia.org/r/474148 (owner: 10Gehel) [11:00:08] (03Abandoned) 10Gehel: Add sonarcloud publication to maven site publish template [integration/config] - 10https://gerrit.wikimedia.org/r/474148 (owner: 10Gehel) [11:04:32] 10MediaWiki-Codesniffer, 10MediaWiki-General-or-Unknown: Enforce usage of phpdoc's @api and @internal tags to indicate public and private structural elements - https://phabricator.wikimedia.org/T209394 (10thiemowmde) The idea presented here sounds nice. However, I am concerned. The relevant piece really is th... [11:04:32] 10MediaWiki-Codesniffer, 10MediaWiki-General-or-Unknown: Enforce usage of phpdoc's @api and @internal tags to indicate public and private structural elements - https://phabricator.wikimedia.org/T209394 (10thiemowmde) The idea presented here sounds nice. However, I am concerned. The relevant piece really is th... [11:07:46] 10MediaWiki-Codesniffer: PHP Codesniffer errors and warnings misleading(?) - https://phabricator.wikimedia.org/T209486 (10thiemowmde) What about "The PHPUnit annotation @group should only be used in comments describing a class or trait"? [11:07:46] 10MediaWiki-Codesniffer: PHP Codesniffer errors and warnings misleading(?) - https://phabricator.wikimedia.org/T209486 (10thiemowmde) What about "The PHPUnit annotation @group should only be used in comments describing a class or trait"? [11:07:54] 10Gerrit, 10Operations: Gerrit is down "502 Proxy Error" - https://phabricator.wikimedia.org/T209456 (10ArielGlenn) [11:07:54] 10Gerrit, 10Operations: Gerrit is down "502 Proxy Error" - https://phabricator.wikimedia.org/T209456 (10ArielGlenn) [11:10:11] maintenance-disconnect-full-disks build 20718 integration-slave-jessie-1002 (/srv: 95%): OFFLINE due to disk space [11:10:11] maintenance-disconnect-full-disks build 20718 integration-slave-jessie-1002 (/srv: 95%): OFFLINE due to disk space [11:13:46] PROBLEM - Free space - all mounts on integration-slave-jessie-1002 is CRITICAL: CRITICAL: integration.integration-slave-jessie-1002.diskspace._mnt.byte_percentfree (No valid datapoints found)integration.integration-slave-jessie-1002.diskspace._srv.byte_percentfree (<20.00%) [11:13:46] PROBLEM - Free space - all mounts on integration-slave-jessie-1002 is CRITICAL: CRITICAL: integration.integration-slave-jessie-1002.diskspace._mnt.byte_percentfree (No valid datapoints found)integration.integration-slave-jessie-1002.diskspace._srv.byte_percentfree (<20.00%) [11:20:10] maintenance-disconnect-full-disks build 20720 integration-slave-jessie-1002: OFFLINE due to disk space [11:20:10] maintenance-disconnect-full-disks build 20720 integration-slave-jessie-1002: OFFLINE due to disk space [11:20:11] maintenance-disconnect-full-disks build 20720 integration-slave-jessie-1003: OFFLINE due to disk space [11:20:11] maintenance-disconnect-full-disks build 20720 integration-slave-jessie-1003: OFFLINE due to disk space [11:26:28] 10Beta-Cluster-Infrastructure, 10Cloud-VPS: Puppet errors on various deployment-prep hosts - https://phabricator.wikimedia.org/T208879 (10fgiunchedi) Fixed `ms-be` and `ms-fe` instances by adding `profile::prometheus::statsd_exporter::mappings: []` to project hiera. [11:26:28] 10Beta-Cluster-Infrastructure, 10Cloud-VPS: Puppet errors on various deployment-prep hosts - https://phabricator.wikimedia.org/T208879 (10fgiunchedi) Fixed `ms-be` and `ms-fe` instances by adding `profile::prometheus::statsd_exporter::mappings: []` to project hiera. [11:33:45] RECOVERY - Free space - all mounts on integration-slave-jessie-1002 is OK: OK: integration.integration-slave-jessie-1002.diskspace._mnt.byte_percentfree (No valid datapoints found) [11:33:45] RECOVERY - Free space - all mounts on integration-slave-jessie-1002 is OK: OK: integration.integration-slave-jessie-1002.diskspace._mnt.byte_percentfree (No valid datapoints found) [11:45:10] maintenance-disconnect-full-disks build 20725 integration-slave-jessie-1002: OFFLINE due to disk space [11:45:10] maintenance-disconnect-full-disks build 20725 integration-slave-jessie-1002: OFFLINE due to disk space [11:45:11] maintenance-disconnect-full-disks build 20725 integration-slave-jessie-1003: OFFLINE due to disk space [11:45:11] maintenance-disconnect-full-disks build 20725 integration-slave-jessie-1003: OFFLINE due to disk space [11:45:46] 10Continuous-Integration-Infrastructure (shipyard), 10Release-Engineering-Team (Kanban), 10Operations, 10decommission, and 2 others: Remove labnodepool1001.eqiad.wmnet - https://phabricator.wikimedia.org/T209642 (10ArielGlenn) p:05Triage>03Normal [11:45:46] 10Continuous-Integration-Infrastructure (shipyard), 10Release-Engineering-Team (Kanban), 10Operations, 10decommission, and 2 others: Remove labnodepool1001.eqiad.wmnet - https://phabricator.wikimedia.org/T209642 (10ArielGlenn) p:05Triage>03Normal [12:10:12] maintenance-disconnect-full-disks build 20730 integration-slave-jessie-1002: OFFLINE due to disk space [12:10:12] maintenance-disconnect-full-disks build 20730 integration-slave-jessie-1002: OFFLINE due to disk space [12:10:13] maintenance-disconnect-full-disks build 20730 integration-slave-jessie-1003: OFFLINE due to disk space [12:10:13] maintenance-disconnect-full-disks build 20730 integration-slave-jessie-1003: OFFLINE due to disk space [12:20:22] Project beta-update-databases-eqiad build #29818: 04FAILURE in 21 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/29818/ [12:20:22] Project beta-update-databases-eqiad build #29818: 04FAILURE in 21 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/29818/ [12:35:11] maintenance-disconnect-full-disks build 20735 integration-slave-jessie-1002: OFFLINE due to disk space [12:35:11] maintenance-disconnect-full-disks build 20735 integration-slave-jessie-1002: OFFLINE due to disk space [12:35:11] maintenance-disconnect-full-disks build 20735 integration-slave-jessie-1003: OFFLINE due to disk space [12:35:11] maintenance-disconnect-full-disks build 20735 integration-slave-jessie-1003: OFFLINE due to disk space [12:42:42] gehel: zeljkof sorry went out for lunch etc [12:42:42] gehel: zeljkof sorry went out for lunch etc [12:43:23] hashar: no problemo :) [12:43:23] hashar: no problemo :) [12:43:34] we just don't know how to update the docker container [12:43:34] we just don't know how to update the docker container [12:43:54] https://gerrit.wikimedia.org/r/c/integration/config/+/472661 [12:43:54] https://gerrit.wikimedia.org/r/c/integration/config/+/472661 [12:44:19] hashar: see https://gerrit.wikimedia.org/r/c/integration/config/+/472661/8..9 [12:44:19] hashar: see https://gerrit.wikimedia.org/r/c/integration/config/+/472661/8..9 [12:45:59] (03CR) 10Hashar: "The docker containers are build using docker-pkg which requires a bump in the changelog file of the container:" (032 comments) [integration/config] - 10https://gerrit.wikimedia.org/r/472661 (https://phabricator.wikimedia.org/T207046) (owner: 10Gehel) [12:45:59] (03CR) 10Hashar: "The docker containers are build using docker-pkg which requires a bump in the changelog file of the container:" (032 comments) [integration/config] - 10https://gerrit.wikimedia.org/r/472661 (https://phabricator.wikimedia.org/T207046) (owner: 10Gehel) [12:46:00] reviewed [12:46:00] reviewed [12:46:19] there is a changelog file that needs to be bumped [12:46:19] there is a changelog file that needs to be bumped [12:46:39] then the containers are build using docker-pkg, a software written by SRE that process the Dockerfile.template and build the containers [12:46:39] then the containers are build using docker-pkg, a software written by SRE that process the Dockerfile.template and build the containers [12:46:46] the build itself is done on contint1001 [12:46:46] the build itself is done on contint1001 [12:47:21] once the change is merged, someone with ssh access to contint1001 can get the containers build using the Phabric task "deploy_docker" [12:47:21] once the change is merged, someone with ssh access to contint1001 can get the containers build using the Phabric task "deploy_docker" [12:47:29] cd projects/integration/config && fab deploy_docker [12:47:29] cd projects/integration/config && fab deploy_docker [12:52:26] hashar: thanks, I'll leave it to gehel to reply, looks like he knows what he's doing :) [12:52:26] hashar: thanks, I'll leave it to gehel to reply, looks like he knows what he's doing :) [13:00:11] maintenance-disconnect-full-disks build 20740 integration-slave-jessie-1002: OFFLINE due to disk space [13:00:11] maintenance-disconnect-full-disks build 20740 integration-slave-jessie-1002: OFFLINE due to disk space [13:00:12] maintenance-disconnect-full-disks build 20740 integration-slave-jessie-1003: OFFLINE due to disk space [13:00:12] maintenance-disconnect-full-disks build 20740 integration-slave-jessie-1003: OFFLINE due to disk space [13:01:14] (03PS10) 10Gehel: Add sonarcloud publication to maven site publish template [integration/config] - 10https://gerrit.wikimedia.org/r/472661 (https://phabricator.wikimedia.org/T207046) [13:01:14] (03PS10) 10Gehel: Add sonarcloud publication to maven site publish template [integration/config] - 10https://gerrit.wikimedia.org/r/472661 (https://phabricator.wikimedia.org/T207046) [13:01:55] hashar: so here is the fix [13:01:55] hashar: so here is the fix [13:02:19] if you have time to deploy the new image, that would be very nice! [13:02:19] if you have time to deploy the new image, that would be very nice! [13:02:23] errand, back in a few [13:02:23] errand, back in a few [13:02:25] gehel: sure [13:02:25] gehel: sure [13:11:22] (03PS11) 10Hashar: docker: set SONA_USER_HOME for SonarQube [integration/config] - 10https://gerrit.wikimedia.org/r/472661 (https://phabricator.wikimedia.org/T207046) (owner: 10Gehel) [13:11:22] (03PS11) 10Hashar: docker: set SONA_USER_HOME for SonarQube [integration/config] - 10https://gerrit.wikimedia.org/r/472661 (https://phabricator.wikimedia.org/T207046) (owner: 10Gehel) [13:11:24] (03PS1) 10Hashar: Add sonarcloud publication to maven site publish template [integration/config] - 10https://gerrit.wikimedia.org/r/474173 (https://phabricator.wikimedia.org/T207046) [13:11:24] (03PS1) 10Hashar: Add sonarcloud publication to maven site publish template [integration/config] - 10https://gerrit.wikimedia.org/r/474173 (https://phabricator.wikimedia.org/T207046) [13:17:38] 10Project-Admins: Phabricator project for an extension LangCodeOverride - https://phabricator.wikimedia.org/T209231 (10Liuxinyu970226) @jeblad Try [[https://www.mediawiki.org/wiki/Gerrit/New_repositories/Requests|Gerrit/New repositories/Requests]] if you aim that this extension should be used by WMF sites? [13:17:38] 10Project-Admins: Phabricator project for an extension LangCodeOverride - https://phabricator.wikimedia.org/T209231 (10Liuxinyu970226) @jeblad Try [[https://www.mediawiki.org/wiki/Gerrit/New_repositories/Requests|Gerrit/New repositories/Requests]] if you aim that this extension should be used by WMF sites? [13:22:32] Yippee, build fixed! [13:22:32] Yippee, build fixed! [13:22:33] Project beta-update-databases-eqiad build #29819: 09FIXED in 2 min 32 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/29819/ [13:22:33] Project beta-update-databases-eqiad build #29819: 09FIXED in 2 min 32 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/29819/ [13:24:11] zeljkof: gehel I have tweaked the change and split them :) [13:24:11] zeljkof: gehel I have tweaked the change and split them :) [13:25:12] maintenance-disconnect-full-disks build 20745 integration-slave-jessie-1002: OFFLINE due to disk space [13:25:12] maintenance-disconnect-full-disks build 20745 integration-slave-jessie-1002: OFFLINE due to disk space [13:25:12] maintenance-disconnect-full-disks build 20745 integration-slave-jessie-1003: OFFLINE due to disk space [13:25:12] maintenance-disconnect-full-disks build 20745 integration-slave-jessie-1003: OFFLINE due to disk space [13:25:58] (03PS12) 10Hashar: docker: set SONA_USER_HOME for SonarQube [integration/config] - 10https://gerrit.wikimedia.org/r/472661 (https://phabricator.wikimedia.org/T207046) (owner: 10Gehel) [13:25:58] (03PS12) 10Hashar: docker: set SONA_USER_HOME for SonarQube [integration/config] - 10https://gerrit.wikimedia.org/r/472661 (https://phabricator.wikimedia.org/T207046) (owner: 10Gehel) [13:26:00] (03PS2) 10Hashar: Add sonarcloud publication to maven site publish template [integration/config] - 10https://gerrit.wikimedia.org/r/474173 (https://phabricator.wikimedia.org/T207046) [13:26:00] (03PS2) 10Hashar: Add sonarcloud publication to maven site publish template [integration/config] - 10https://gerrit.wikimedia.org/r/474173 (https://phabricator.wikimedia.org/T207046) [13:27:59] (03CR) 10Hashar: docker: set SONA_USER_HOME for SonarQube (033 comments) [integration/config] - 10https://gerrit.wikimedia.org/r/472661 (https://phabricator.wikimedia.org/T207046) (owner: 10Gehel) [13:27:59] (03CR) 10Hashar: docker: set SONA_USER_HOME for SonarQube (033 comments) [integration/config] - 10https://gerrit.wikimedia.org/r/472661 (https://phabricator.wikimedia.org/T207046) (owner: 10Gehel) [13:28:06] (03CR) 10Hashar: [C: 032] docker: set SONA_USER_HOME for SonarQube [integration/config] - 10https://gerrit.wikimedia.org/r/472661 (https://phabricator.wikimedia.org/T207046) (owner: 10Gehel) [13:28:06] (03CR) 10Hashar: [C: 032] docker: set SONA_USER_HOME for SonarQube [integration/config] - 10https://gerrit.wikimedia.org/r/472661 (https://phabricator.wikimedia.org/T207046) (owner: 10Gehel) [13:28:24] gehel: zeljkof I am deploying the containers [13:28:24] gehel: zeljkof I am deploying the containers [13:28:38] I am not sure whether all repositories actually support sonar:sonar though! [13:28:38] I am not sure whether all repositories actually support sonar:sonar though! [13:29:15] (03CR) 10jerkins-bot: [V: 04-1] Add sonarcloud publication to maven site publish template [integration/config] - 10https://gerrit.wikimedia.org/r/474173 (https://phabricator.wikimedia.org/T207046) (owner: 10Hashar) [13:29:15] (03CR) 10jerkins-bot: [V: 04-1] Add sonarcloud publication to maven site publish template [integration/config] - 10https://gerrit.wikimedia.org/r/474173 (https://phabricator.wikimedia.org/T207046) (owner: 10Hashar) [13:29:49] (03CR) 10Hashar: "recheck" [integration/config] - 10https://gerrit.wikimedia.org/r/474173 (https://phabricator.wikimedia.org/T207046) (owner: 10Hashar) [13:29:50] (03CR) 10Hashar: "recheck" [integration/config] - 10https://gerrit.wikimedia.org/r/474173 (https://phabricator.wikimedia.org/T207046) (owner: 10Hashar) [13:31:14] (03Merged) 10jenkins-bot: docker: set SONA_USER_HOME for SonarQube [integration/config] - 10https://gerrit.wikimedia.org/r/472661 (https://phabricator.wikimedia.org/T207046) (owner: 10Gehel) [13:31:14] (03Merged) 10jenkins-bot: docker: set SONA_USER_HOME for SonarQube [integration/config] - 10https://gerrit.wikimedia.org/r/472661 (https://phabricator.wikimedia.org/T207046) (owner: 10Gehel) [13:32:56] ! [13:32:56] ! [13:33:09] !log building docker image releng/java8:0.4.1 for https://gerrit.wikimedia.org/r/#/c/integration/config/+/472661/ [13:33:09] !log building docker image releng/java8:0.4.1 for https://gerrit.wikimedia.org/r/#/c/integration/config/+/472661/ [13:33:11] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [13:33:11] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [13:34:21] hashar: cool! [13:34:21] hashar: cool! [13:34:37] gehel: let me know when the patch is ready, and I can deploy the jobs, so we can test it now [13:34:37] gehel: let me know when the patch is ready, and I can deploy the jobs, so we can test it now [13:39:16] [contint1001.wikimedia.org] out: docker.errors.BuildError: The command '/bin/sh -c echo 'Acquire::http::Proxy "http://webproxy.eqiad.wmnet:8080";' > /etc/apt/apt.conf.d/80_proxy && apt-get update && DEBIAN_FRONTEND=noninteractive apt-get install --yes gpg wget --no-install-recommends && rm -f /etc/apt/apt.conf.d/80_proxy && apt-get clean && rm -rf /var/lib/apt/lists/* && cd /tmp && wget http://www.apache.org/dist/ [13:39:16] maven/maven-3/3.5.2/binaries/apache-maven-3.5.2-bin.tar.gz && gpg --import /tmp/KEYS && gpg --verify apache-maven-3.5.2-bin.tar.gz.asc && tar -C /opt -zxf apache-maven-3.5.2-bin.tar.gz && apt purge --yes gpg wget && rm -rf ~/.gnupg' returned a non-zero code: 8 [13:39:16] [contint1001.wikimedia.org] out: docker.errors.BuildError: The command '/bin/sh -c echo 'Acquire::http::Proxy "http://webproxy.eqiad.wmnet:8080";' > /etc/apt/apt.conf.d/80_proxy && apt-get update && DEBIAN_FRONTEND=noninteractive apt-get install --yes gpg wget --no-install-recommends && rm -f /etc/apt/apt.conf.d/80_proxy && apt-get clean && rm -rf /var/lib/apt/lists/* && cd /tmp && wget http://www.apache.org/dist/ [13:39:16] maven/maven-3/3.5.2/binaries/apache-maven-3.5.2-bin.tar.gz && gpg --import /tmp/KEYS && gpg --verify apache-maven-3.5.2-bin.tar.gz.asc && tar -C /opt -zxf apache-maven-3.5.2-bin.tar.gz && apt purge --yes gpg wget && rm -rf ~/.gnupg' returned a non-zero code: 8 [13:39:20] the container does not build :/ [13:39:20] the container does not build :/ [13:50:11] maintenance-disconnect-full-disks build 20750 integration-slave-jessie-1002: OFFLINE due to disk space [13:50:11] maintenance-disconnect-full-disks build 20750 integration-slave-jessie-1002: OFFLINE due to disk space [13:50:12] maintenance-disconnect-full-disks build 20750 integration-slave-jessie-1003: OFFLINE due to disk space [13:50:12] maintenance-disconnect-full-disks build 20750 integration-slave-jessie-1003: OFFLINE due to disk space [13:57:22] zeljkof: https://gerrit.wikimedia.org/r/c/search/extra/+/473763 ready to merge [13:57:22] zeljkof: https://gerrit.wikimedia.org/r/c/search/extra/+/473763 ready to merge [13:58:40] how, just saw hashar's message about the container not building :( [13:58:40] how, just saw hashar's message about the container not building :( [13:59:16] any idea why? [13:59:16] any idea why? [13:59:58] container not building? [13:59:58] container not building? [14:00:02] I've missed that one [14:00:02] I've missed that one [14:00:14] na [14:00:14] na [14:00:19] gotta find out what is wrong in it :((( [14:00:19] gotta find out what is wrong in it :((( [14:00:31] hashar: good luck! [14:00:31] hashar: good luck! [14:00:36] hashar: so wait, we can test this or not? [14:00:36] hashar: so wait, we can test this or not? [14:00:55] gehel: the jobs are deployed, if the container is there, you can merge your commit [14:00:55] gehel: the jobs are deployed, if the container is there, you can merge your commit [14:01:11] but I'm confused, I'm not sure if the relevant container is updated in CI [14:01:11] but I'm confused, I'm not sure if the relevant container is updated in CI [14:01:27] * gehel is just as confused:) [14:01:27] ACTION is just as confused:) [14:02:47] gehel: well, you can make the merge, and we'll see :D [14:02:48] gehel: well, you can make the merge, and we'll see :D [14:03:12] ah, now I see [14:03:12] ah, now I see [14:03:17] > the container does not build :/ [14:03:18] > the container does not build :/ [14:03:29] ok, so I guess we'll wait [14:03:29] ok, so I guess we'll wait [14:03:41] I'll deploy the jobs back to normal, so they are not broken [14:03:41] I'll deploy the jobs back to normal, so they are not broken [14:03:52] * zeljkof back in 5 minutes [14:03:52] ACTION back in 5 minutes [14:05:13] maintenance-disconnect-full-disks build 20753 integration-slave-jessie-1001 (/srv: 99%): OFFLINE due to disk space [14:05:14] maintenance-disconnect-full-disks build 20753 integration-slave-jessie-1001 (/srv: 99%): OFFLINE due to disk space [14:15:10] maintenance-disconnect-full-disks build 20755 integration-slave-jessie-1001: OFFLINE due to disk space [14:15:10] maintenance-disconnect-full-disks build 20755 integration-slave-jessie-1001: OFFLINE due to disk space [14:15:11] maintenance-disconnect-full-disks build 20755 integration-slave-jessie-1002: OFFLINE due to disk space [14:15:11] maintenance-disconnect-full-disks build 20755 integration-slave-jessie-1002: OFFLINE due to disk space [14:15:11] maintenance-disconnect-full-disks build 20755 integration-slave-jessie-1003: OFFLINE due to disk space [14:15:11] maintenance-disconnect-full-disks build 20755 integration-slave-jessie-1003: OFFLINE due to disk space [14:18:06] grrr [14:18:06] grrr [14:18:10] I gotta run it manually [14:18:10] I gotta run it manually [14:19:17] --2018-11-16 14:18:55-- http://www.apache.org/dist/maven/maven-3/3.5.2/binaries/apache-maven-3.5.2-bin.tar.gz [14:19:17] --2018-11-16 14:18:55-- http://www.apache.org/dist/maven/maven-3/3.5.2/binaries/apache-maven-3.5.2-bin.tar.gz [14:19:22] 2018-11-16 14:18:55 ERROR 404: Not Found. [14:19:22] 2018-11-16 14:18:55 ERROR 404: Not Found. [14:20:11] not sure why integration-slave-jessie-10x have issues [14:20:11] not sure why integration-slave-jessie-10x have issues [14:21:44] 10Continuous-Integration-Infrastructure, 10Operations, 10Traffic: trafficserver debian-glue builds failing on integration-slave-jessie-1001: No space left on device - https://phabricator.wikimedia.org/T209703 (10ema) [14:21:44] 10Continuous-Integration-Infrastructure, 10Operations, 10Traffic: trafficserver debian-glue builds failing on integration-slave-jessie-1001: No space left on device - https://phabricator.wikimedia.org/T209703 (10ema) [14:22:00] 10Continuous-Integration-Infrastructure, 10Operations, 10Traffic: trafficserver debian-glue builds failing on integration-slave-jessie-1001: No space left on device - https://phabricator.wikimedia.org/T209703 (10ema) p:05Triage>03Normal [14:22:00] 10Continuous-Integration-Infrastructure, 10Operations, 10Traffic: trafficserver debian-glue builds failing on integration-slave-jessie-1001: No space left on device - https://phabricator.wikimedia.org/T209703 (10ema) p:05Triage>03Normal [14:23:46] hashar: it looks like only the latest minor releases are kept [14:23:46] hashar: it looks like only the latest minor releases are kept [14:23:59] 3.5.2 does not seem to be downloadable anymore [14:23:59] 3.5.2 does not seem to be downloadable anymore [14:24:04] or at least not from that URL [14:24:04] or at least not from that URL [14:26:13] so we gotta bump the maven version :) [14:26:13] so we gotta bump the maven version :) [14:26:19] hashar: https://repo.maven.apache.org/maven2/org/apache/maven/apache-maven/3.5.2/apache-maven-3.5.2-bin.tar.gz is probably a more stable URL [14:26:19] hashar: https://repo.maven.apache.org/maven2/org/apache/maven/apache-maven/3.5.2/apache-maven-3.5.2-bin.tar.gz is probably a more stable URL [14:26:54] but it should not hurt to upgrade to 3.6 [14:26:54] but it should not hurt to upgrade to 3.6 [14:27:15] and one more reason to use mvnw :) [14:27:15] and one more reason to use mvnw :) [14:30:02] (03PS1) 10Hashar: docker: update maven URL for releng/java8 [integration/config] - 10https://gerrit.wikimedia.org/r/474192 [14:30:02] (03PS1) 10Hashar: docker: update maven URL for releng/java8 [integration/config] - 10https://gerrit.wikimedia.org/r/474192 [14:30:08] gehel: ^ :) [14:30:08] gehel: ^ :) [14:31:04] (03CR) 10Gehel: [C: 031] "LGTM" [integration/config] - 10https://gerrit.wikimedia.org/r/474192 (owner: 10Hashar) [14:31:04] (03CR) 10Gehel: [C: 031] "LGTM" [integration/config] - 10https://gerrit.wikimedia.org/r/474192 (owner: 10Hashar) [14:31:20] (03PS3) 10Hashar: Add sonarcloud publication to maven site publish template [integration/config] - 10https://gerrit.wikimedia.org/r/474173 (https://phabricator.wikimedia.org/T207046) [14:31:20] (03PS3) 10Hashar: Add sonarcloud publication to maven site publish template [integration/config] - 10https://gerrit.wikimedia.org/r/474173 (https://phabricator.wikimedia.org/T207046) [14:31:34] (03CR) 10Hashar: [C: 032] docker: update maven URL for releng/java8 [integration/config] - 10https://gerrit.wikimedia.org/r/474192 (owner: 10Hashar) [14:31:34] (03CR) 10Hashar: [C: 032] docker: update maven URL for releng/java8 [integration/config] - 10https://gerrit.wikimedia.org/r/474192 (owner: 10Hashar) [14:31:49] then I have to fix up my ruby bundler installation :( [14:31:49] then I have to fix up my ruby bundler installation :( [14:32:17] fixed! [14:32:17] fixed! [14:32:26] that was fast! [14:32:26] that was fast! [14:33:20] (03Merged) 10jenkins-bot: docker: update maven URL for releng/java8 [integration/config] - 10https://gerrit.wikimedia.org/r/474192 (owner: 10Hashar) [14:33:20] (03Merged) 10jenkins-bot: docker: update maven URL for releng/java8 [integration/config] - 10https://gerrit.wikimedia.org/r/474192 (owner: 10Hashar) [14:33:58] 10Continuous-Integration-Infrastructure (shipyard), 10Release-Engineering-Team (Kanban), 10Operations, 10decommission, and 2 others: Remove labnodepool1001.eqiad.wmnet - https://phabricator.wikimedia.org/T209642 (10Dzahn) >>! In T209642#4753118, @aborrero wrote: > Do we have info on specs and expiration d... [14:33:58] 10Continuous-Integration-Infrastructure (shipyard), 10Release-Engineering-Team (Kanban), 10Operations, 10decommission, and 2 others: Remove labnodepool1001.eqiad.wmnet - https://phabricator.wikimedia.org/T209642 (10Dzahn) >>! In T209642#4753118, @aborrero wrote: > Do we have info on specs and expiration d... [14:35:26] !log Building docker image releng/java8:0.4.2 [14:35:26] !log Building docker image releng/java8:0.4.2 [14:35:28] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [14:35:28] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [14:39:18] 10Continuous-Integration-Infrastructure (shipyard), 10Release-Engineering-Team (Kanban), 10Operations, 10decommission, and 2 others: Remove labnodepool1001.eqiad.wmnet - https://phabricator.wikimedia.org/T209642 (10faidon) This specific HW is /very/ old and is already overdue for decomissioning (by 3 years... [14:39:18] 10Continuous-Integration-Infrastructure (shipyard), 10Release-Engineering-Team (Kanban), 10Operations, 10decommission, and 2 others: Remove labnodepool1001.eqiad.wmnet - https://phabricator.wikimedia.org/T209642 (10faidon) This specific HW is /very/ old and is already overdue for decomissioning (by 3 years... [14:43:48] gpg: failed to start agent '/usr/bin/gpg-agent': No such file or directory [14:43:48] gpg: failed to start agent '/usr/bin/gpg-agent': No such file or directory [14:45:19] gpg: cannot open '/dev/tty': No such device or address [14:45:19] gpg: cannot open '/dev/tty': No such device or address [14:45:20] pff [14:45:20] pff [14:50:21] (03PS4) 10Hashar: Add sonarcloud publication to maven site publish template [integration/config] - 10https://gerrit.wikimedia.org/r/474173 (https://phabricator.wikimedia.org/T207046) [14:50:21] (03PS4) 10Hashar: Add sonarcloud publication to maven site publish template [integration/config] - 10https://gerrit.wikimedia.org/r/474173 (https://phabricator.wikimedia.org/T207046) [14:50:24] (03PS1) 10Hashar: docker: fix gpg in releng/java8 [integration/config] - 10https://gerrit.wikimedia.org/r/474270 [14:50:24] (03PS1) 10Hashar: docker: fix gpg in releng/java8 [integration/config] - 10https://gerrit.wikimedia.org/r/474270 [14:53:11] gehel: I think the image managed to be build [14:53:11] gehel: I think the image managed to be build [14:55:58] gehel: zeljkof the new container should be ready. I have rebased the jjb change to use the latest version of releng/java8 [14:55:58] gehel: zeljkof the new container should be ready. I have rebased the jjb change to use the latest version of releng/java8 [14:56:14] hashar: thanks! [14:56:14] hashar: thanks! [14:56:27] * gehel is waiting to merge a last test change [14:56:27] ACTION is waiting to merge a last test change [14:56:34] hashar: thanks! [14:56:34] hashar: thanks! [14:56:48] gehel: I just need a minute to finish something, and I'll deploy the jobs [14:56:48] gehel: I just need a minute to finish something, and I'll deploy the jobs [15:00:15] gehel: jobs deployed! go ahead and merge the commit [15:00:15] gehel: jobs deployed! go ahead and merge the commit [15:01:22] I've +2d it [15:01:22] I've +2d it [15:01:45] done [15:01:45] done [15:02:30] ok, so ci is super busy [15:02:30] ok, so ci is super busy [15:02:40] looks like we'll have to wati [15:02:40] looks like we'll have to wati [15:02:41] wait [15:02:41] wait [15:03:41] looks like ci is not that busy :) [15:03:41] looks like ci is not that busy :) [15:03:43] https://integration.wikimedia.org/ci/job/search-extra-maven-java8-docker-site-publish/31/console [15:03:43] https://integration.wikimedia.org/ci/job/search-extra-maven-java8-docker-site-publish/31/console [15:06:46] hm, it's stuck [15:06:46] hm, it's stuck [15:06:52] `rsync: failed to set times on "/cache/.": Operation not permitted (1)` [15:06:52] `rsync: failed to set times on "/cache/.": Operation not permitted (1)` [15:06:56] maybe that's normal? [15:06:56] maybe that's normal? [15:07:42] that one is seen everytime [15:07:42] that one is seen everytime [15:10:21] (03CR) 10jerkins-bot: [V: 04-1] Add sonarcloud publication to maven site publish template [integration/config] - 10https://gerrit.wikimedia.org/r/474173 (https://phabricator.wikimedia.org/T207046) (owner: 10Hashar) [15:10:21] (03CR) 10jerkins-bot: [V: 04-1] Add sonarcloud publication to maven site publish template [integration/config] - 10https://gerrit.wikimedia.org/r/474173 (https://phabricator.wikimedia.org/T207046) (owner: 10Hashar) [15:13:30] sonar plugins are loading, we're at least one step further than before! [15:13:30] sonar plugins are loading, we're at least one step further than before! [15:15:12] zeljkof: `rsync: failed to set times on "/cache/.": Operation not permitted (1)` <--- yes that is "normal" [15:15:12] zeljkof: `rsync: failed to set times on "/cache/.": Operation not permitted (1)` <--- yes that is "normal" [15:15:13] gehel: nothing broke! [15:15:13] gehel: nothing broke! [15:15:16] zeljkof: the message is harmless [15:15:16] zeljkof: the message is harmless [15:16:02] the /cache directory is the directory on the host system , rsync attempts to set some date on the directory and it is obviously not possible to change the host dir from within the container [15:16:02] the /cache directory is the directory on the host system , rsync attempts to set some date on the directory and it is obviously not possible to change the host dir from within the container [15:16:15] analysis has been published: https://sonarcloud.io/dashboard?id=org.wikimedia.search%3Aextra-parent [15:16:15] analysis has been published: https://sonarcloud.io/dashboard?id=org.wikimedia.search%3Aextra-parent [15:16:19] [INFO] ANALYSIS SUCCESSFUL, you can browse https://sonarcloud.io/dashboard?id=org.wikimedia.search%3Aextra-parent [15:16:19] [INFO] ANALYSIS SUCCESSFUL, you can browse https://sonarcloud.io/dashboard?id=org.wikimedia.search%3Aextra-parent [15:18:54] hashar, zeljkof : Thanks a bunch! [15:18:54] hashar, zeljkof : Thanks a bunch! [15:18:55] gehel: can we see the most recent upload to https://sonarcloud.io/dashboard?id=org.wikimedia.search%3Aextra-parent [15:18:55] gehel: can we see the most recent upload to https://sonarcloud.io/dashboard?id=org.wikimedia.search%3Aextra-parent [15:19:10] yes, there is an activity timeline on the right [15:19:10] yes, there is an activity timeline on the right [15:19:30] https://www.irccloud.com/pastebin/ajDkphhP/ [15:19:30] https://www.irccloud.com/pastebin/ajDkphhP/ [15:19:42] ok, so it works? [15:19:42] ok, so it works? [15:19:47] it does! [15:19:47] it does! [15:20:04] ok, then I'll merge the patch in gerrit, the jobs are already deployed [15:20:04] ok, then I'll merge the patch in gerrit, the jobs are already deployed [15:20:56] argh, but the patch failed CI :D [15:20:56] argh, but the patch failed CI :D [15:20:57] https://gerrit.wikimedia.org/r/c/integration/config/+/474173 [15:20:57] https://gerrit.wikimedia.org/r/c/integration/config/+/474173 [15:21:22] every step is fighting us! [15:21:22] every step is fighting us! [15:22:02] I think it's just a ci problem, one job is aborted [15:22:02] I think it's just a ci problem, one job is aborted [15:22:04] I're recheck [15:22:04] I're recheck [15:22:11] (03CR) 10Zfilipin: "recheck" [integration/config] - 10https://gerrit.wikimedia.org/r/474173 (https://phabricator.wikimedia.org/T207046) (owner: 10Hashar) [15:22:11] (03CR) 10Zfilipin: "recheck" [integration/config] - 10https://gerrit.wikimedia.org/r/474173 (https://phabricator.wikimedia.org/T207046) (owner: 10Hashar) [15:22:46] gehel: zeljkof quite nice, kudos :) [15:22:46] gehel: zeljkof quite nice, kudos :) [15:23:02] hashar: thanks for all the help! [15:23:02] hashar: thanks for all the help! [15:23:07] yeah, thanks both! [15:23:07] yeah, thanks both! [15:23:16] next step: do something with all those info! [15:23:17] next step: do something with all those info! [15:23:20] can we just mvn sonar:sonar on any repo? [15:23:20] can we just mvn sonar:sonar on any repo? [15:23:30] almost [15:23:31] almost [15:23:42] well, it has to be a maven project [15:23:42] well, it has to be a maven project [15:23:51] [ERROR] Failed to execute goal org.sonarsource.scanner.maven:sonar-maven-plugin:3.5.0.1254:sonar (default-cli): Goal requires a project to execute but there is no POM in this directory (/home/hashar/projects/integration/config). Please verify you invoked Maven from the correct directory. -> [Help 1] [15:23:51] [ERROR] Failed to execute goal org.sonarsource.scanner.maven:sonar-maven-plugin:3.5.0.1254:sonar (default-cli): Goal requires a project to execute but there is no POM in this directory (/home/hashar/projects/integration/config). Please verify you invoked Maven from the correct directory. -> [Help 1] [15:23:52] bah [15:23:52] bah [15:23:52] and it needs minimal configuration [15:23:52] and it needs minimal configuration [15:24:12] for other projects, you'll need the sonar scanner [15:24:12] for other projects, you'll need the sonar scanner [15:24:25] maybe we can craft a stock sonar job that would come with a default pom.xml [15:24:26] maybe we can craft a stock sonar job that would come with a default pom.xml [15:24:31] I have no idea how it works really :) [15:24:31] I have no idea how it works really :) [15:24:41] better to use the sonar scanner [15:24:42] better to use the sonar scanner [15:25:16] bringing all the complexity of maven just to run an analyzer does not look like a great idea [15:25:16] bringing all the complexity of maven just to run an analyzer does not look like a great idea [15:25:50] but we might want to create a docker image containing the sonar analyzer [15:25:50] but we might want to create a docker image containing the sonar analyzer [15:25:55] should not be too hard [15:25:55] should not be too hard [15:26:08] ok, ci is happy with the patch, I'll merge it [15:26:08] ok, ci is happy with the patch, I'll merge it [15:26:14] (03CR) 10Zfilipin: [C: 032] Add sonarcloud publication to maven site publish template [integration/config] - 10https://gerrit.wikimedia.org/r/474173 (https://phabricator.wikimedia.org/T207046) (owner: 10Hashar) [15:26:14] (03CR) 10Zfilipin: [C: 032] Add sonarcloud publication to maven site publish template [integration/config] - 10https://gerrit.wikimedia.org/r/474173 (https://phabricator.wikimedia.org/T207046) (owner: 10Hashar) [15:26:49] damn, sonar even has a lua analyzer [15:26:49] damn, sonar even has a lua analyzer [15:29:26] (03CR) 10jerkins-bot: [V: 04-1] Add sonarcloud publication to maven site publish template [integration/config] - 10https://gerrit.wikimedia.org/r/474173 (https://phabricator.wikimedia.org/T207046) (owner: 10Hashar) [15:29:26] (03CR) 10jerkins-bot: [V: 04-1] Add sonarcloud publication to maven site publish template [integration/config] - 10https://gerrit.wikimedia.org/r/474173 (https://phabricator.wikimedia.org/T207046) (owner: 10Hashar) [15:36:52] 10Release-Engineering-Team (Kanban), 10User-zeljkofilipin: Find top 15 target projects that could use Selenium tests to prevent incidents - https://phabricator.wikimedia.org/T199133 (10zeljkofilipin) [15:36:52] 10Release-Engineering-Team (Kanban), 10User-zeljkofilipin: Find top 15 target projects that could use Selenium tests to prevent incidents - https://phabricator.wikimedia.org/T199133 (10zeljkofilipin) [15:38:55] 10Release-Engineering-Team (Kanban), 10Cloud-Services, 10Epic: Migrate the Integration cloud project to eqiad1-r - https://phabricator.wikimedia.org/T208803 (10Andrew) I've moved several docker-XXXX jessie-XXXX nodes and that seems to work fine. So I'll finish up with those on my own. The next VMs of inter... [15:38:55] 10Release-Engineering-Team (Kanban), 10Cloud-Services, 10Epic: Migrate the Integration cloud project to eqiad1-r - https://phabricator.wikimedia.org/T208803 (10Andrew) I've moved several docker-XXXX jessie-XXXX nodes and that seems to work fine. So I'll finish up with those on my own. The next VMs of inter... [15:42:20] (03PS1) 10Hashar: Explain why integration-publishing IP is hardcoded [integration/config] - 10https://gerrit.wikimedia.org/r/474279 (https://phabricator.wikimedia.org/T208803) [15:42:21] (03PS1) 10Hashar: Explain why integration-publishing IP is hardcoded [integration/config] - 10https://gerrit.wikimedia.org/r/474279 (https://phabricator.wikimedia.org/T208803) [15:47:03] 10Release-Engineering-Team (Kanban), 10Cloud-Services, 10Epic, 10Patch-For-Review: Migrate the Integration cloud project to eqiad1-r - https://phabricator.wikimedia.org/T208803 (10hashar) So for integration-publishing, its used from contint1001 which is a production machine. The server thus can not resolve... [15:47:04] 10Release-Engineering-Team (Kanban), 10Cloud-Services, 10Epic, 10Patch-For-Review: Migrate the Integration cloud project to eqiad1-r - https://phabricator.wikimedia.org/T208803 (10hashar) So for integration-publishing, its used from contint1001 which is a production machine. The server thus can not resolve... [15:57:12] waiting for a vote from jenkins-bot since a rebase 12 minutes ago. related to move from nodepool to docker? [15:57:12] waiting for a vote from jenkins-bot since a rebase 12 minutes ago. related to move from nodepool to docker? [15:57:42] ok, i heard about the maintenance and backlog now.. ok [15:57:42] ok, i heard about the maintenance and backlog now.. ok [16:14:41] (03CR) 10Andrew Bogott: [C: 032] Explain why integration-publishing IP is hardcoded [integration/config] - 10https://gerrit.wikimedia.org/r/474279 (https://phabricator.wikimedia.org/T208803) (owner: 10Hashar) [16:14:41] (03CR) 10Andrew Bogott: [C: 032] Explain why integration-publishing IP is hardcoded [integration/config] - 10https://gerrit.wikimedia.org/r/474279 (https://phabricator.wikimedia.org/T208803) (owner: 10Hashar) [16:36:06] (03PS1) 10D3r1ck01: Add PageImages extension dependency to MobileFrontend [integration/config] - 10https://gerrit.wikimedia.org/r/474290 [16:36:06] (03PS1) 10D3r1ck01: Add PageImages extension dependency to MobileFrontend [integration/config] - 10https://gerrit.wikimedia.org/r/474290 [16:42:53] 10Gerrit, 10Operations: Gerrit is down "502 Proxy Error" - https://phabricator.wikimedia.org/T209456 (10thcipriani) 05Open>03Resolved a:03thcipriani Two days later -- after adding in some better monitoring and blocking bots from indexing git blame for giant files with a lot of history -- we seem to be in... [16:42:53] 10Gerrit, 10Operations: Gerrit is down "502 Proxy Error" - https://phabricator.wikimedia.org/T209456 (10thcipriani) 05Open>03Resolved a:03thcipriani Two days later -- after adding in some better monitoring and blocking bots from indexing git blame for giant files with a lot of history -- we seem to be in... [16:54:15] (03PS2) 10Takidelfin: Jenkins whitelist: Add takidelfin [GCI] [integration/config] - 10https://gerrit.wikimedia.org/r/473671 (https://phabricator.wikimedia.org/T200778) [16:54:15] (03PS2) 10Takidelfin: Jenkins whitelist: Add takidelfin [GCI] [integration/config] - 10https://gerrit.wikimedia.org/r/473671 (https://phabricator.wikimedia.org/T200778) [17:19:05] !log Added two more Jenkins docker slaves: https://integration.wikimedia.org/ci/computer/integration-slave-docker-1044/ and https://integration.wikimedia.org/ci/computer/integration-slave-docker-1045/ [17:19:05] !log Added two more Jenkins docker slaves: https://integration.wikimedia.org/ci/computer/integration-slave-docker-1044/ and https://integration.wikimedia.org/ci/computer/integration-slave-docker-1045/ [17:19:07] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [17:19:07] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [17:20:42] !log Deleted integration-slave-docker-1045 (wrong OS stretch, must be jessie) [17:20:42] !log Deleted integration-slave-docker-1045 (wrong OS stretch, must be jessie) [17:20:44] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [17:20:44] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [17:23:39] 10Release-Engineering-Team, 10Epic, 10Patch-For-Review: Migrate all CI jobs from Nodepool, deprecate its use - https://phabricator.wikimedia.org/T190097 (10greg) >>! In T190097#4751523, @hashar wrote: > All jobs have been migrated. There are still some sub tasks opened but they are no more blockers for phasi... [17:23:39] 10Release-Engineering-Team, 10Epic, 10Patch-For-Review: Migrate all CI jobs from Nodepool, deprecate its use - https://phabricator.wikimedia.org/T190097 (10greg) >>! In T190097#4751523, @hashar wrote: > All jobs have been migrated. There are still some sub tasks opened but they are no more blockers for phasi... [17:35:43] 10Continuous-Integration-Infrastructure (shipyard), 10Release-Engineering-Team (Kanban), 10Operations, 10Patch-For-Review, 10cloud-services-team (Kanban): Phase out Nodepool from production - https://phabricator.wikimedia.org/T209361 (10hashar) [17:35:43] 10Continuous-Integration-Infrastructure (shipyard), 10Release-Engineering-Team (Kanban), 10Operations, 10Patch-For-Review, 10cloud-services-team (Kanban): Phase out Nodepool from production - https://phabricator.wikimedia.org/T209361 (10hashar) [17:37:09] !log Added integration-slave-docker-1046 and integration-slave-docker-1047 [17:37:09] !log Added integration-slave-docker-1046 and integration-slave-docker-1047 [17:37:11] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [17:37:11] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [17:40:26] legoktm: for when you are back. LibraryUpgrader has enqueued a lot of changes 15:00 UTC . That was me gracefully shutting down zuul for a config change. I guess the Library upgrader kept enqueuing change [17:40:26] legoktm: for when you are back. LibraryUpgrader has enqueued a lot of changes 15:00 UTC . That was me gracefully shutting down zuul for a config change. I guess the Library upgrader kept enqueuing change [17:40:48] eventually the queue of changes got drained in a couple hours [17:40:48] eventually the queue of changes got drained in a couple hours [17:41:11] all is fine. not much complaints received :) [17:41:11] all is fine. not much complaints received :) [17:41:18] and I have added 3 new slaves [17:41:18] and I have added 3 new slaves [17:41:26] that is all. I am on week-end. Have a good friday everyone [17:41:26] that is all. I am on week-end. Have a good friday everyone [18:05:35] (03CR) 10D3r1ck01: "recheck" [integration/config] - 10https://gerrit.wikimedia.org/r/473671 (https://phabricator.wikimedia.org/T200778) (owner: 10Takidelfin) [18:05:35] (03CR) 10D3r1ck01: "recheck" [integration/config] - 10https://gerrit.wikimedia.org/r/473671 (https://phabricator.wikimedia.org/T200778) (owner: 10Takidelfin) [18:07:57] more slaves makes me happy [18:07:57] more slaves makes me happy [18:09:09] (03PS1) 10Bartosz Dziewoński: Fix whitelist email for Flutterish [integration/config] - 10https://gerrit.wikimedia.org/r/474306 [18:09:09] (03PS1) 10Bartosz Dziewoński: Fix whitelist email for Flutterish [integration/config] - 10https://gerrit.wikimedia.org/r/474306 [18:09:28] RECOVERY - English Wikipedia Main page on beta-cluster is OK: HTTP OK: HTTP/1.1 200 OK - 47964 bytes in 1.448 second response time [18:09:29] RECOVERY - English Wikipedia Main page on beta-cluster is OK: HTTP OK: HTTP/1.1 200 OK - 47964 bytes in 1.448 second response time [18:09:29] RECOVERY - App Server Main HTTP Response on deployment-mediawiki-07 is OK: HTTP OK: HTTP/1.1 200 OK - 47345 bytes in 1.423 second response time [18:09:29] RECOVERY - App Server Main HTTP Response on deployment-mediawiki-07 is OK: HTTP OK: HTTP/1.1 200 OK - 47345 bytes in 1.423 second response time [18:09:29] PROBLEM - Puppet staleness on deployment-rd3-cptest-master01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [43200.0] [18:09:29] PROBLEM - Puppet staleness on deployment-rd3-cptest-master01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [43200.0] [18:09:29] RECOVERY - English Wikipedia Mobile Main page on beta-cluster is OK: HTTP OK: HTTP/1.1 200 OK - 36858 bytes in 1.581 second response time [18:09:29] RECOVERY - English Wikipedia Mobile Main page on beta-cluster is OK: HTTP OK: HTTP/1.1 200 OK - 36858 bytes in 1.581 second response time [18:09:30] PROBLEM - Long lived cherry-picks on puppetmaster on deployment-puppetmaster03 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [18:09:30] PROBLEM - Long lived cherry-picks on puppetmaster on deployment-puppetmaster03 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [18:10:23] hi, anyone around to approve https://gerrit.wikimedia.org/r/c/integration/config/+/474306 for me? [18:10:23] hi, anyone around to approve https://gerrit.wikimedia.org/r/c/integration/config/+/474306 for me? [18:10:55] (03CR) 10jerkins-bot: [V: 04-1] Jenkins whitelist: Add takidelfin [GCI] [integration/config] - 10https://gerrit.wikimedia.org/r/473671 (https://phabricator.wikimedia.org/T200778) (owner: 10Takidelfin) [18:10:55] (03CR) 10jerkins-bot: [V: 04-1] Jenkins whitelist: Add takidelfin [GCI] [integration/config] - 10https://gerrit.wikimedia.org/r/473671 (https://phabricator.wikimedia.org/T200778) (owner: 10Takidelfin) [18:11:39] (03CR) 10D3r1ck01: "recheck" [integration/config] - 10https://gerrit.wikimedia.org/r/473671 (https://phabricator.wikimedia.org/T200778) (owner: 10Takidelfin) [18:11:39] (03CR) 10D3r1ck01: "recheck" [integration/config] - 10https://gerrit.wikimedia.org/r/473671 (https://phabricator.wikimedia.org/T200778) (owner: 10Takidelfin) [18:13:11] (03CR) 10D3r1ck01: "So this patch is just meant for testing a patch on MF extension that depends on PageImages extension." [integration/config] - 10https://gerrit.wikimedia.org/r/474290 (owner: 10D3r1ck01) [18:13:11] (03CR) 10D3r1ck01: "So this patch is just meant for testing a patch on MF extension that depends on PageImages extension." [integration/config] - 10https://gerrit.wikimedia.org/r/474290 (owner: 10D3r1ck01) [18:13:47] (03PS2) 10D3r1ck01: [DNM] Add PageImages extension dependency to MobileFrontend [integration/config] - 10https://gerrit.wikimedia.org/r/474290 [18:13:47] (03PS2) 10D3r1ck01: [DNM] Add PageImages extension dependency to MobileFrontend [integration/config] - 10https://gerrit.wikimedia.org/r/474290 [18:15:43] (03CR) 10jerkins-bot: [V: 04-1] Fix whitelist email for Flutterish [integration/config] - 10https://gerrit.wikimedia.org/r/474306 (owner: 10Bartosz Dziewoński) [18:15:43] (03CR) 10jerkins-bot: [V: 04-1] Fix whitelist email for Flutterish [integration/config] - 10https://gerrit.wikimedia.org/r/474306 (owner: 10Bartosz Dziewoński) [18:39:21] MatmaRex: yes, I can do so in a bit [18:39:21] MatmaRex: yes, I can do so in a bit [18:56:48] 10Release-Engineering-Team (Kanban), 10User-greg: Onboarding Jeena Huneidi - https://phabricator.wikimedia.org/T209722 (10greg) p:05Triage>03Normal [18:56:48] 10Release-Engineering-Team (Kanban), 10User-greg: Onboarding Jeena Huneidi - https://phabricator.wikimedia.org/T209722 (10greg) p:05Triage>03Normal [19:12:02] legoktm: thanks [19:12:02] legoktm: thanks [19:27:07] 10Gerrit: Gerrit upload-pack send ALL references causing massive network I/O on common operations - https://phabricator.wikimedia.org/T103990 (10Paladox) [19:27:07] 10Gerrit: Gerrit upload-pack send ALL references causing massive network I/O on common operations - https://phabricator.wikimedia.org/T103990 (10Paladox) [19:27:11] 10Gerrit, 10Patch-For-Review: Upgrade to Gerrit 2.16 - https://phabricator.wikimedia.org/T200739 (10Paladox) [19:27:12] 10Gerrit, 10Patch-For-Review: Upgrade to Gerrit 2.16 - https://phabricator.wikimedia.org/T200739 (10Paladox) [19:28:42] 10Gerrit, 10Patch-For-Review: Upgrade to Gerrit 2.16 - https://phabricator.wikimedia.org/T200739 (10Paladox) [19:28:42] 10Gerrit, 10Patch-For-Review: Upgrade to Gerrit 2.16 - https://phabricator.wikimedia.org/T200739 (10Paladox) [19:57:17] (03CR) 10Legoktm: "recheck" [integration/config] - 10https://gerrit.wikimedia.org/r/474306 (owner: 10Bartosz Dziewoński) [19:57:17] (03CR) 10Legoktm: "recheck" [integration/config] - 10https://gerrit.wikimedia.org/r/474306 (owner: 10Bartosz Dziewoński) [20:00:27] (03CR) 10jerkins-bot: [V: 04-1] Fix whitelist email for Flutterish [integration/config] - 10https://gerrit.wikimedia.org/r/474306 (owner: 10Bartosz Dziewoński) [20:00:27] (03CR) 10jerkins-bot: [V: 04-1] Fix whitelist email for Flutterish [integration/config] - 10https://gerrit.wikimedia.org/r/474306 (owner: 10Bartosz Dziewoński) [20:01:46] CI tests for the CI are failing. what are we ever gonna dooooo [20:01:46] CI tests for the CI are failing. what are we ever gonna dooooo [20:04:49] (03CR) 10Bartosz Dziewoński: "(not a real failure)" [integration/config] - 10https://gerrit.wikimedia.org/r/474306 (owner: 10Bartosz Dziewoński) [20:04:49] (03CR) 10Bartosz Dziewoński: "(not a real failure)" [integration/config] - 10https://gerrit.wikimedia.org/r/474306 (owner: 10Bartosz Dziewoński) [20:08:12] lol [20:08:12] lol [20:31:02] greg-g: Is the week on Nov 26 a regular deployment week? There's nothing for this week written under https://wikitech.wikimedia.org/wiki/Deployments#Upcoming. [20:31:02] greg-g: Is the week on Nov 26 a regular deployment week? There's nothing for this week written under https://wikitech.wikimedia.org/wiki/Deployments#Upcoming. [20:32:05] bearND: sorry, yes it's normal, it's on my list to make that week's entry today. I'm a day behind :) [20:32:05] bearND: sorry, yes it's normal, it's on my list to make that week's entry today. I'm a day behind :) [20:32:32] greg-g: That's cool. Thanks for the confirmation. [20:32:32] greg-g: That's cool. Thanks for the confirmation. [20:33:33] (03CR) 10Legoktm: [C: 032] Fix whitelist email for Flutterish [integration/config] - 10https://gerrit.wikimedia.org/r/474306 (owner: 10Bartosz Dziewoński) [20:33:33] (03CR) 10Legoktm: [C: 032] Fix whitelist email for Flutterish [integration/config] - 10https://gerrit.wikimedia.org/r/474306 (owner: 10Bartosz Dziewoński) [20:35:18] (03Merged) 10jenkins-bot: Fix whitelist email for Flutterish [integration/config] - 10https://gerrit.wikimedia.org/r/474306 (owner: 10Bartosz Dziewoński) [20:35:18] (03Merged) 10jenkins-bot: Fix whitelist email for Flutterish [integration/config] - 10https://gerrit.wikimedia.org/r/474306 (owner: 10Bartosz Dziewoński) [20:35:20] (03PS1) 10Legoktm: Bump timeout of integration-config-tox-docker to 5 minutes [integration/config] - 10https://gerrit.wikimedia.org/r/474326 [20:35:20] (03PS1) 10Legoktm: Bump timeout of integration-config-tox-docker to 5 minutes [integration/config] - 10https://gerrit.wikimedia.org/r/474326 [20:35:40] (03CR) 10Legoktm: [C: 032] Bump timeout of integration-config-tox-docker to 5 minutes [integration/config] - 10https://gerrit.wikimedia.org/r/474326 (owner: 10Legoktm) [20:35:41] (03CR) 10Legoktm: [C: 032] Bump timeout of integration-config-tox-docker to 5 minutes [integration/config] - 10https://gerrit.wikimedia.org/r/474326 (owner: 10Legoktm) [20:36:07] !log deployed https://gerrit.wikimedia.org/r/474306 [20:36:07] !log deployed https://gerrit.wikimedia.org/r/474306 [20:36:09] MatmaRex: ^^ [20:36:09] MatmaRex: ^^ [20:36:09] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [20:36:09] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [20:36:12] greg-g: do you have any thoughts about how I can move forward on T208101? I can keep doing piecemeal instance-by-instance moves but that just means twice as much downtime. [20:36:12] greg-g: do you have any thoughts about how I can move forward on T208101? I can keep doing piecemeal instance-by-instance moves but that just means twice as much downtime. [20:36:12] T208101: Migrate deployment-prep to eqiad1 - https://phabricator.wikimedia.org/T208101 [20:36:13] T208101: Migrate deployment-prep to eqiad1 - https://phabricator.wikimedia.org/T208101 [20:36:29] thanks legoktm [20:36:29] thanks legoktm [20:37:39] (03CR) 10Legoktm: [C: 04-1] Jenkins whitelist: Add takidelfin [GCI] (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/473671 (https://phabricator.wikimedia.org/T200778) (owner: 10Takidelfin) [20:37:39] (03CR) 10Legoktm: [C: 04-1] Jenkins whitelist: Add takidelfin [GCI] (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/473671 (https://phabricator.wikimedia.org/T200778) (owner: 10Takidelfin) [20:39:06] (03Merged) 10jenkins-bot: Bump timeout of integration-config-tox-docker to 5 minutes [integration/config] - 10https://gerrit.wikimedia.org/r/474326 (owner: 10Legoktm) [20:39:06] (03Merged) 10jenkins-bot: Bump timeout of integration-config-tox-docker to 5 minutes [integration/config] - 10https://gerrit.wikimedia.org/r/474326 (owner: 10Legoktm) [20:40:54] (03PS3) 10Legoktm: Jenkins whitelist: Add takidelfin [GCI] [integration/config] - 10https://gerrit.wikimedia.org/r/473671 (https://phabricator.wikimedia.org/T200778) (owner: 10Takidelfin) [20:40:54] (03PS3) 10Legoktm: Jenkins whitelist: Add takidelfin [GCI] [integration/config] - 10https://gerrit.wikimedia.org/r/473671 (https://phabricator.wikimedia.org/T200778) (owner: 10Takidelfin) [20:40:57] (03PS1) 10Legoktm: Add missing $ to regexes in whitelist [integration/config] - 10https://gerrit.wikimedia.org/r/474330 [20:40:57] (03PS1) 10Legoktm: Add missing $ to regexes in whitelist [integration/config] - 10https://gerrit.wikimedia.org/r/474330 [20:44:56] https://integration.wikimedia.org/ci/job/integration-config-tox-docker/buildTimeTrend [20:44:56] https://integration.wikimedia.org/ci/job/integration-config-tox-docker/buildTimeTrend [20:45:14] something very clearly changed to increase the runtime [20:45:15] something very clearly changed to increase the runtime [20:46:31] (03CR) 10Legoktm: [C: 032] Jenkins whitelist: Add takidelfin [GCI] [integration/config] - 10https://gerrit.wikimedia.org/r/473671 (https://phabricator.wikimedia.org/T200778) (owner: 10Takidelfin) [20:46:31] (03CR) 10Legoktm: [C: 032] Jenkins whitelist: Add takidelfin [GCI] [integration/config] - 10https://gerrit.wikimedia.org/r/473671 (https://phabricator.wikimedia.org/T200778) (owner: 10Takidelfin) [20:46:34] (03CR) 10Legoktm: [C: 032] Add missing $ to regexes in whitelist [integration/config] - 10https://gerrit.wikimedia.org/r/474330 (owner: 10Legoktm) [20:46:34] (03CR) 10Legoktm: [C: 032] Add missing $ to regexes in whitelist [integration/config] - 10https://gerrit.wikimedia.org/r/474330 (owner: 10Legoktm) [20:46:41] legoktm: I've moved a couple of the worker nodes, which means that you've been running at n-1 nodes for much of the last 24 hours. I don't know if that's adequate to explain the issue [20:46:41] legoktm: I've moved a couple of the worker nodes, which means that you've been running at n-1 nodes for much of the last 24 hours. I don't know if that's adequate to explain the issue [20:47:00] (um, and honestly I don't know for sure that the nodes I moved are related to that exact graph) [20:47:00] (um, and honestly I don't know for sure that the nodes I moved are related to that exact graph) [20:48:23] andrewbogott: hm, theoretically reduced availability of nodes shouldn't affect how long the job runs, just how long it waits in queue. [20:48:23] andrewbogott: hm, theoretically reduced availability of nodes shouldn't affect how long the job runs, just how long it waits in queue. [20:48:41] but I've also seen weird stuff where jenkins consistently schedules jobs on a specific node, and that node is busted in some way perf wise [20:48:41] but I've also seen weird stuff where jenkins consistently schedules jobs on a specific node, and that node is busted in some way perf wise [20:48:42] agreed [20:48:42] agreed [20:48:50] if the graph doesn't include wait time [20:48:50] if the graph doesn't include wait time [20:49:05] it doesn't, just run time [20:49:05] it doesn't, just run time [20:50:36] (03Merged) 10jenkins-bot: Jenkins whitelist: Add takidelfin [GCI] [integration/config] - 10https://gerrit.wikimedia.org/r/473671 (https://phabricator.wikimedia.org/T200778) (owner: 10Takidelfin) [20:50:37] (03Merged) 10jenkins-bot: Jenkins whitelist: Add takidelfin [GCI] [integration/config] - 10https://gerrit.wikimedia.org/r/473671 (https://phabricator.wikimedia.org/T200778) (owner: 10Takidelfin) [20:50:39] (03Merged) 10jenkins-bot: Add missing $ to regexes in whitelist [integration/config] - 10https://gerrit.wikimedia.org/r/474330 (owner: 10Legoktm) [20:50:39] (03Merged) 10jenkins-bot: Add missing $ to regexes in whitelist [integration/config] - 10https://gerrit.wikimedia.org/r/474330 (owner: 10Legoktm) [20:51:09] !log deploying https://gerrit.wikimedia.org/r/473671 https://gerrit.wikimedia.org/r/474330 [20:51:09] !log deploying https://gerrit.wikimedia.org/r/473671 https://gerrit.wikimedia.org/r/474330 [20:51:11] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [20:51:11] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [20:57:07] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team, 10Cloud-Services, 10Epic: Migrate deployment-prep to eqiad1 - https://phabricator.wikimedia.org/T208101 (10Andrew) So, by the time y'all are back from your offsite I will need to be powering down labvirt1010 and 1011 (they are leased and their leas... [20:57:07] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team, 10Cloud-Services, 10Epic: Migrate deployment-prep to eqiad1 - https://phabricator.wikimedia.org/T208101 (10Andrew) So, by the time y'all are back from your offsite I will need to be powering down labvirt1010 and 1011 (they are leased and their leas... [20:58:45] (03CR) 10Legoktm: [C: 032] Add mogmog123 to jenkins whitelist (GCI 2018 student). [integration/config] - 10https://gerrit.wikimedia.org/r/473718 (https://phabricator.wikimedia.org/T200778) (owner: 10Mogmog123) [20:58:45] (03CR) 10Legoktm: [C: 032] Add mogmog123 to jenkins whitelist (GCI 2018 student). [integration/config] - 10https://gerrit.wikimedia.org/r/473718 (https://phabricator.wikimedia.org/T200778) (owner: 10Mogmog123) [20:59:27] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team, 10Operations: contint1001 store docker images on separate partition or disk - https://phabricator.wikimedia.org/T207707 (10thcipriani) @RobH would you know if it's possible to add physical storage to this machine? If not we'll have to work... [20:59:27] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team, 10Operations: contint1001 store docker images on separate partition or disk - https://phabricator.wikimedia.org/T207707 (10thcipriani) @RobH would you know if it's possible to add physical storage to this machine? If not we'll have to work... [20:59:38] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team, 10Cloud-Services, 10Epic: Migrate deployment-prep to eqiad1 - https://phabricator.wikimedia.org/T208101 (10mmodell) @andrew: Can it be done next week? I know that's pushing it given that it's a 3 day work week. [20:59:38] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team, 10Cloud-Services, 10Epic: Migrate deployment-prep to eqiad1 - https://phabricator.wikimedia.org/T208101 (10mmodell) @andrew: Can it be done next week? I know that's pushing it given that it's a 3 day work week. [20:59:41] How far away are we from running the CI jobs without a whitelist? [20:59:41] How far away are we from running the CI jobs without a whitelist? [21:00:20] I noted the nodepool task got resolved, yay. ;-) I vaguely recall moving to docker being a necessary but not sufficient condition/ [21:00:20] I noted the nodepool task got resolved, yay. ;-) I vaguely recall moving to docker being a necessary but not sufficient condition/ [21:00:28] (03Merged) 10jenkins-bot: Add mogmog123 to jenkins whitelist (GCI 2018 student). [integration/config] - 10https://gerrit.wikimedia.org/r/473718 (https://phabricator.wikimedia.org/T200778) (owner: 10Mogmog123) [21:00:29] (03Merged) 10jenkins-bot: Add mogmog123 to jenkins whitelist (GCI 2018 student). [integration/config] - 10https://gerrit.wikimedia.org/r/473718 (https://phabricator.wikimedia.org/T200778) (owner: 10Mogmog123) [21:01:21] nodepool was actually the first attempt at removing the whitelist.... [21:01:21] nodepool was actually the first attempt at removing the whitelist.... [21:01:28] James_F: hash mentioned this yesterday. With nodepool gone we still have to phase out a couple physical hosts that are still lingering, but I don't know the steps past that one. [21:01:28] James_F: hash mentioned this yesterday. With nodepool gone we still have to phase out a couple physical hosts that are still lingering, but I don't know the steps past that one. [21:01:29] !log deploying https://gerrit.wikimedia.org/r/473718 [21:01:29] !log deploying https://gerrit.wikimedia.org/r/473718 [21:01:31] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [21:01:31] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [21:01:50] greg-g: Indeed. [21:01:50] greg-g: Indeed. [21:02:02] * James_F nods. [21:02:02] ACTION nods. [21:02:31] Is there a task for "run CI for all"? [21:02:31] Is there a task for "run CI for all"? [21:02:44] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team, 10Operations: contint1001 store docker images on separate partition or disk - https://phabricator.wikimedia.org/T207707 (10RobH) contint1001 was purchased on T130738, and has dual 1TB SATA disks. We generally don't store anything in the /... [21:02:44] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team, 10Operations: contint1001 store docker images on separate partition or disk - https://phabricator.wikimedia.org/T207707 (10RobH) contint1001 was purchased on T130738, and has dual 1TB SATA disks. We generally don't store anything in the /... [21:03:01] I think someone needs to look at the threat models, and what exactly does "arbitrary code access in a docker instance runing on a cloud vps instance" let you escalate to [21:03:02] I think someone needs to look at the threat models, and what exactly does "arbitrary code access in a docker instance runing on a cloud vps instance" let you escalate to [21:03:34] Ah, yes, T192217. [21:03:34] Ah, yes, T192217. [21:03:34] T192217: Remove the "check" pipeline and Zuul's user-filter - https://phabricator.wikimedia.org/T192217 [21:03:34] T192217: Remove the "check" pipeline and Zuul's user-filter - https://phabricator.wikimedia.org/T192217 [21:03:37] and then realize that it doesn't matter since we already let basically everyone do so [21:03:37] and then realize that it doesn't matter since we already let basically everyone do so [21:04:47] maybe a bit of preventative stuff like closing outbound mail ports and other network firewalling [21:04:47] maybe a bit of preventative stuff like closing outbound mail ports and other network firewalling [21:05:15] Would it be valuable to put this into that task (or somewhere else)? [21:05:15] Would it be valuable to put this into that task (or somewhere else)? [21:06:27] +1 [21:06:28] +1 [21:07:22] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team: Remove the "check" pipeline and Zuul's user-filter - https://phabricator.wikimedia.org/T192217 (10Legoktm) Some notes from IRC (lightly edited): ` [12:59:41] How far away are we from running the CI jobs without a whitelist? [13:00:2... [21:07:22] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team: Remove the "check" pipeline and Zuul's user-filter - https://phabricator.wikimedia.org/T192217 (10Legoktm) Some notes from IRC (lightly edited): ` [12:59:41] How far away are we from running the CI jobs without a whitelist? [13:00:2... [21:07:41] ❤️ [21:07:41] ❤️ [21:11:57] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team, 10Cloud-Services, 10Epic: Migrate deployment-prep to eqiad1 - https://phabricator.wikimedia.org/T208101 (10Andrew) Yep, next week would work just fine for me. I'm available until at least Wednesday evening, and can multi-task if it runs longer tha... [21:11:57] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team, 10Cloud-Services, 10Epic: Migrate deployment-prep to eqiad1 - https://phabricator.wikimedia.org/T208101 (10Andrew) Yep, next week would work just fine for me. I'm available until at least Wednesday evening, and can multi-task if it runs longer tha... [21:26:33] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team: Remove the "check" pipeline and Zuul's user-filter - https://phabricator.wikimedia.org/T192217 (10EddieGP) > [13:03:01] I think someone needs to look at the threat models, and what exactly does "arbitrary code access in a docker ins... [21:26:33] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team: Remove the "check" pipeline and Zuul's user-filter - https://phabricator.wikimedia.org/T192217 (10EddieGP) > [13:03:01] I think someone needs to look at the threat models, and what exactly does "arbitrary code access in a docker ins... [22:07:03] https://gerrit-review.googlesource.com/c/gerrit/+/204968 ! [22:07:03] https://gerrit-review.googlesource.com/c/gerrit/+/204968 ! [22:07:53] 10Continuous-Integration-Config, 10StructuredDiscussions, 10User-kostajh, 10Growth-Team (Current Sprint): Random test failure Flow\Tests\Api\ApiFlowModerateTopicTest::testModerateLockedTopic and Flow\Tests\Collection\PostCollectionTest::testGetLastRevision - https://phabricator.wikimedia.org/T209729 (10D3r1... [22:07:53] 10Continuous-Integration-Config, 10StructuredDiscussions, 10User-kostajh, 10Growth-Team (Current Sprint): Random test failure Flow\Tests\Api\ApiFlowModerateTopicTest::testModerateLockedTopic and Flow\Tests\Collection\PostCollectionTest::testGetLastRevision - https://phabricator.wikimedia.org/T209729 (10D3r1... [22:20:41] 10Continuous-Integration-Config, 10StructuredDiscussions, 10User-kostajh, 10Growth-Team (Current Sprint): Random test failure Flow\Tests\Api\ApiFlowModerateTopicTest::testModerateLockedTopic and Flow\Tests\Collection\PostCollectionTest::testGetLastRevision - https://phabricator.wikimedia.org/T209729 (10kost... [22:20:41] 10Continuous-Integration-Config, 10StructuredDiscussions, 10User-kostajh, 10Growth-Team (Current Sprint): Random test failure Flow\Tests\Api\ApiFlowModerateTopicTest::testModerateLockedTopic and Flow\Tests\Collection\PostCollectionTest::testGetLastRevision - https://phabricator.wikimedia.org/T209729 (10kost...