[00:13:19] RECOVERY - Puppet errors on integration-slave-docker-1001 is OK: OK: Less than 1.00% above the threshold [0.0] [00:16:49] Yippee, build fixed! [00:16:49] Project selenium-Flow » chrome,beta,Linux,BrowserTests build #525: 09FIXED in 48 sec: https://integration.wikimedia.org/ci/job/selenium-Flow/BROWSER=chrome,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=BrowserTests/525/ [01:33:20] PROBLEM - Puppet staleness on deployment-kafka-jumbo-1 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [43200.0] [03:23:42] PROBLEM - English Wikipedia Mobile Main page on beta-cluster is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:28:35] RECOVERY - English Wikipedia Mobile Main page on beta-cluster is OK: HTTP OK: HTTP/1.1 200 OK - 35279 bytes in 1.018 second response time [03:39:16] PROBLEM - Puppet errors on integration-slave-docker-1001 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [04:14:18] RECOVERY - Puppet errors on integration-slave-docker-1001 is OK: OK: Less than 1.00% above the threshold [0.0] [04:58:50] PROBLEM - Mediawiki Error Rate on graphite-labs is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [10.0] [04:59:21] 10Release-Engineering-Team, 10Cleanup, 10Repository-Admins, 10Patch-For-Review, 10User-MarcoAurelio: Archive Extension:Phalanx - https://phabricator.wikimedia.org/T176665#3633249 (10SamanthaNguyen) @MarcoAurelio: Thanks for getting around on this; I've been meaning to do it but I haven't so I appreciate... [05:38:48] PROBLEM - Mediawiki Error Rate on graphite-labs is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [10.0] [06:40:17] PROBLEM - Puppet errors on integration-slave-docker-1001 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [07:45:18] RECOVERY - Puppet errors on integration-slave-docker-1001 is OK: OK: Less than 1.00% above the threshold [0.0] [08:03:48] RECOVERY - Mediawiki Error Rate on graphite-labs is OK: OK: Less than 1.00% above the threshold [1.0] [08:06:19] PROBLEM - Puppet errors on integration-slave-docker-1001 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [08:36:39] 10Continuous-Integration-Config, 10Release-Engineering-Team (Kanban): CI job debian-glue-non-voting: add support for BACKPORTS=yes - https://phabricator.wikimedia.org/T173999#3634696 (10hashar) 05Resolved>03Open I am not sure what is going on since the hook on the machine looks like: ``` name=/var/cache/p... [08:57:12] hashar: I'm going to go for the zuul reload with that low-prio queue patch [08:57:40] Thoughts on adding my email to the list now so I can check if its working and removing in a subsequent patch once confirmed? [09:02:25] (03PS4) 10Addshore: Low prio queue for libraryupdater [integration/config] - 10https://gerrit.wikimedia.org/r/380307 [09:02:39] (03PS5) 10Addshore: Low prio queue for libraryupdater [integration/config] - 10https://gerrit.wikimedia.org/r/380307 [09:15:02] Hey releng, there is a question for you on Tech/News talk page: "Is so a frequent release of new mediawiki software really needed?" https://meta.wikimedia.org/w/index.php?title=Talk:Tech/News&diff=0&oldid=17241446 [09:16:17] RECOVERY - Puppet errors on integration-slave-docker-1001 is OK: OK: Less than 1.00% above the threshold [0.0] [09:29:02] 10Continuous-Integration-Config, 10Release-Engineering-Team (Kanban), 10Patch-For-Review: CI job debian-glue-non-voting: add support for BACKPORTS=yes - https://phabricator.wikimedia.org/T173999#3634795 (10hashar) 05Open>03Resolved So `BACKPORTS` was shallowed by `sudo`. I have adjusted the sudo policy i... [09:50:09] (03CR) 10Addshore: [C: 032] Add AdvancedSearch to make-wmf-branch [tools/release] - 10https://gerrit.wikimedia.org/r/379710 (owner: 10Addshore) [09:50:35] (03CR) 10Addshore: [C: 032] Remove .* from .gitignore [integration/config] - 10https://gerrit.wikimedia.org/r/380553 (owner: 10Hashar) [09:50:47] (03Merged) 10jenkins-bot: Add AdvancedSearch to make-wmf-branch [tools/release] - 10https://gerrit.wikimedia.org/r/379710 (owner: 10Addshore) [09:51:39] (03Merged) 10jenkins-bot: Remove .* from .gitignore [integration/config] - 10https://gerrit.wikimedia.org/r/380553 (owner: 10Hashar) [09:56:29] (03CR) 10Addshore: dockerfile: port build script to python (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/379775 (owner: 10Hashar) [10:00:23] (03CR) 10Addshore: [C: 04-1] "I'm actually rather pro bash, as it is less likely to keep breaking cross platform." [integration/config] - 10https://gerrit.wikimedia.org/r/379775 (owner: 10Hashar) [10:04:24] 10Continuous-Integration-Infrastructure (shipyard), 10WMDE-Analytics-Engineering, 10Patch-For-Review, 10User-Addshore: Have CI run lintr for analytics/wmde/WDCM R files - https://phabricator.wikimedia.org/T176194#3634845 (10Addshore) [10:06:08] addshore> Im actually rather pro bash, as it is less likely to keep breaking cross platform. [10:06:22] or even node, hah [10:06:24] hehe [10:06:30] python is just the worst [10:06:53] even a php script would be less likely to start farting on windows every times it is changes [10:07:12] well [10:07:15] it is either PHP or me [10:07:16] :D [10:07:17] PROBLEM - Puppet errors on integration-slave-docker-1001 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [10:07:25] I choose you! :D [10:07:42] so from https://gerrit.wikimedia.org/r/#/c/379775/ [10:07:48] looks like the failure comes from docker build itself ? [10:07:59] unable to prepare context: unable to evaluate symlinks in Dockerfile path: GetFileAttributesEx C:\Users\adam\dev\git\gerrit\integration\config\dockerfiles\lintr\Dockerfile: The system cannot find the file specified. [10:08:54] why would it consider it a symlink :( [10:09:55] or it cant find the file [10:11:02] so, I pass in paths like //c/foo/bar into docker build [10:11:20] I think this would also be solved by using a python library to talk to docker instead of using commands directly [10:11:23] there must be one [10:11:34] ah yeah for sure [10:11:34] and that would also allow us to run the thing in docker as welll :P dockerception! [10:11:44] I just blindly converted the shell script to a python script [10:11:52] it might actually work if I run it in powershell :/ [10:12:17] do you actually have that integration\config\dockerfiles\lintr\Dockerfile file ? [10:12:21] or is that a symlink? [10:13:01] yeh, i actually have that [10:13:22] * hashar suspects Docker doesn't support links on Windows file systems [10:13:24] oh wait [10:13:24] no [10:13:31] because thats not merged yet [10:13:38] * hashar grabs a coffeee [10:14:07] python build.py tox works [10:14:43] (03CR) 10Addshore: [C: 032] "Looks like the error was because the DockerFile actually doesn't exist :D" [integration/config] - 10https://gerrit.wikimedia.org/r/379775 (owner: 10Hashar) [10:14:47] hashar: ^^ [10:15:46] (03Merged) 10jenkins-bot: dockerfile: port build script to python [integration/config] - 10https://gerrit.wikimedia.org/r/379775 (owner: 10Hashar) [10:17:42] oh man [10:17:43] ! [10:17:57] though on looking at the code, we dont validate that the given image names actually have dockerfile [10:18:01] but heck that will be for later [10:21:12] it seems to break for images that do have a prebuild.sh [10:21:40] https://www.irccloud.com/pastebin/fQ3U2BaS/ [10:22:37] (03PS2) 10Hashar: dockerfiles: allow prebuild.sh without executable bit [integration/config] - 10https://gerrit.wikimedia.org/r/379297 [10:23:04] sweet [10:23:09] (03PS3) 10Hashar: dockerfiles: allow prebuild.sh without executable bit [integration/config] - 10https://gerrit.wikimedia.org/r/379297 [10:23:30] bah that would be a path issue I guess [10:23:42] (03CR) 10Addshore: [C: 032] dockerfiles: allow prebuild.sh without executable bit [integration/config] - 10https://gerrit.wikimedia.org/r/379297 (owner: 10Hashar) [10:23:51] na works even from a parent task [10:23:55] hashar: no, the executable bit patch fixes it [10:24:01] the composer/prebuild.sh lacked the executable bit at some point [10:24:08] but legoktm fixed it in some patch [10:24:15] well, windows doesnt have a concept of executable bits anyway [10:24:17] sooooo :p [10:24:25] ;D [10:25:07] (03Merged) 10jenkins-bot: dockerfiles: allow prebuild.sh without executable bit [integration/config] - 10https://gerrit.wikimedia.org/r/379297 (owner: 10Hashar) [10:25:40] (03PS1) 10Addshore: Docker: build.py Remove any temporary cache-buster files [integration/config] - 10https://gerrit.wikimedia.org/r/380719 [10:25:42] hashar: ^^ :D [10:26:05] (03Abandoned) 10Addshore: Docker: build.sh Remove any temporary cache-buster files [integration/config] - 10https://gerrit.wikimedia.org/r/378807 (owner: 10Addshore) [10:27:37] (03Abandoned) 10Hashar: dockerfiles: pass shellcheck [integration/config] - 10https://gerrit.wikimedia.org/r/379300 (owner: 10Hashar) [10:28:04] addshore: yeah looking at it [10:28:37] (03CR) 10Addshore: [C: 04-1] docker: clone puppet.git in a different layer (032 comments) [integration/config] - 10https://gerrit.wikimedia.org/r/378911 (owner: 10Hashar) [10:29:05] (03CR) 10Addshore: [C: 031] Typo fix in docs [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/380637 (owner: 10MarcoAurelio) [10:29:51] (03CR) 10jerkins-bot: [V: 04-1] Docker: build.py Remove any temporary cache-buster files [integration/config] - 10https://gerrit.wikimedia.org/r/380719 (owner: 10Addshore) [10:31:22] i find it so hard to read the tox failure results [10:32:35] (03CR) 10Hashar: "lovely! :)" (032 comments) [integration/config] - 10https://gerrit.wikimedia.org/r/380719 (owner: 10Addshore) [10:32:38] finally ./dockerfiles/build.py:10:1: F811 redefinition of unused 'glob' from line 5 [10:32:41] addshore: some python ideas ^^ [10:32:53] yeah cause I already: from glob import glob [10:32:58] so your import glob is redundant [10:33:01] or something like that [10:33:25] also map() is quite nice (same as php array_map) [10:33:35] ooooh, okay, thats quite nice, well done python [10:33:36] but feel free to keep the for f in x [10:34:42] (03PS2) 10Addshore: Docker: build.py Remove any temporary cache-buster files [integration/config] - 10https://gerrit.wikimedia.org/r/380719 [10:35:23] but I guess map would not execute all the functions if there is some error [10:35:46] (03PS1) 10Addshore: docker: wrap docker commands in try block [integration/config] - 10https://gerrit.wikimedia.org/r/380720 [10:36:08] 10Release-Engineering-Team, 10Cleanup, 10Repository-Admins, 10Patch-For-Review, 10User-MarcoAurelio: Archive Extension:Phalanx - https://phabricator.wikimedia.org/T176665#3634925 (10MarcoAurelio) >>! In T176665#3634235, @SamanthaNguyen wrote: > @MarcoAurelio: Thanks for getting around on this; I've been... [10:36:36] (03CR) 10Hashar: [C: 04-1] "recheck" [integration/config] - 10https://gerrit.wikimedia.org/r/379507 (owner: 10Hashar) [10:36:43] (03CR) 10jerkins-bot: [V: 04-1] dockerfiles: support for a build.env file and http_proxy [integration/config] - 10https://gerrit.wikimedia.org/r/379507 (owner: 10Hashar) [10:37:17] 10Release-Engineering-Team, 10Cleanup, 10Repository-Admins, 10User-MarcoAurelio: Archive Extension:Phalanx - https://phabricator.wikimedia.org/T176665#3634926 (10MarcoAurelio) [10:37:21] (03PS4) 10Addshore: docker: lintr image - linter for R [integration/config] - 10https://gerrit.wikimedia.org/r/378831 (https://phabricator.wikimedia.org/T176194) [10:37:28] 10Release-Engineering-Team, 10Cleanup, 10Repository-Admins, 10User-MarcoAurelio: Archive Extension:Phalanx - https://phabricator.wikimedia.org/T176665#3633249 (10MarcoAurelio) [10:37:37] addshore: hmm. Actually we could move the try: further up before the prebuild.sh [10:37:49] in case prebuild.sh creates a cache buster file and later fail for some reason [10:38:01] sounds good! [10:38:03] notably, it does: git ls-remote > cache-buster [10:38:11] so the cache-buster is created by bash (and would be empty) [10:38:23] if ls-remote fails, you are left with an empty cache-buster file [10:38:28] (sorry I am being picky [10:38:40] 10Release-Engineering-Team, 10Cleanup, 10Repository-Admins, 10User-MarcoAurelio: Archive Extension:Phalanx - https://phabricator.wikimedia.org/T176665#3633249 (10MarcoAurelio) [10:38:47] (03PS2) 10Addshore: docker: wrap docker commands in try block [integration/config] - 10https://gerrit.wikimedia.org/r/380720 [10:39:00] (03CR) 10Hashar: [C: 032] Docker: build.py Remove any temporary cache-buster files [integration/config] - 10https://gerrit.wikimedia.org/r/380719 (owner: 10Addshore) [10:39:11] 10Beta-Cluster-Infrastructure, 10Multimedia, 10Thumbor, 10Multimedia-Team-Working-Board: On beta commons, thumbnailing of 3D files is broken still - https://phabricator.wikimedia.org/T170444#3634930 (10Cparle) @matthiasmullie this is still in "code review" on the Multimedia-Team-Working-Board ... shouldn't... [10:39:16] 10Release-Engineering-Team, 10Cleanup, 10Repository-Admins, 10User-MarcoAurelio: Archive Extension:Phalanx - https://phabricator.wikimedia.org/T176665#3633249 (10MarcoAurelio) Okay so when the GitHub mirror gets deleted and the gerrit repo is marked back as read-only this should be all done. [10:39:28] (03CR) 10Hashar: [C: 032] docker: wrap docker commands in try block [integration/config] - 10https://gerrit.wikimedia.org/r/380720 (owner: 10Addshore) [10:39:33] \O/ [10:44:09] (03Merged) 10jenkins-bot: Docker: build.py Remove any temporary cache-buster files [integration/config] - 10https://gerrit.wikimedia.org/r/380719 (owner: 10Addshore) [10:44:15] (03Merged) 10jenkins-bot: docker: wrap docker commands in try block [integration/config] - 10https://gerrit.wikimedia.org/r/380720 (owner: 10Addshore) [10:50:26] (03PS5) 10Addshore: docker: lintr image - linter for R [integration/config] - 10https://gerrit.wikimedia.org/r/378831 (https://phabricator.wikimedia.org/T176194) [10:51:05] (03CR) 10GoranSMilovanovic: [C: 031] docker: lintr image - linter for R [integration/config] - 10https://gerrit.wikimedia.org/r/378831 (https://phabricator.wikimedia.org/T176194) (owner: 10Addshore) [10:51:33] !loig docker push docker.io/wmfreleng/lintr:v2017.09.26.10.49 & latest (PS5 of https://gerrit.wikimedia.org/r/#/c/378831/) [10:52:21] (03PS3) 10Addshore: Add experimental lintr job for analytics/wmde/WDCM [integration/config] - 10https://gerrit.wikimedia.org/r/379818 (https://phabricator.wikimedia.org/T176194) [10:53:00] (03CR) 10GoranSMilovanovic: [C: 031] Add experimental lintr job for analytics/wmde/WDCM [integration/config] - 10https://gerrit.wikimedia.org/r/379818 (https://phabricator.wikimedia.org/T176194) (owner: 10Addshore) [10:55:46] hashar: whats publishers are there? [10:55:49] and how do they work? xD [11:01:53] I guess i need to make on for output like https://integration.wikimedia.org/ci/job/lintr-docker/lastSuccessfulBuild/artifact/log/lintr.log/*view*/ :D [11:01:57] addshore: they run after the builders have run [11:02:12] so in theory the build is always a SUCCESS (though it can FAIL) [11:02:33] then the publishers would archive logs / look at test results and eventually set the build from SUCESS to UNSTABLE [11:02:46] yeah [11:02:53] - archive-log-dir or something like that [11:03:07] in jjb/macro.yaml [11:03:21] that would grab anything under $WORKSPACE/log/ [11:04:18] coool [11:04:45] I gotta cook something && eat [11:06:03] me too :D [11:37:21] hashar: I could make a publisher for the lintr thing if I had the text-finder plugin [11:37:40] unless there is a way to build publishers that decide if a build has failed using just bash, which I imagine there is [11:47:20] RECOVERY - Puppet errors on integration-slave-docker-1001 is OK: OK: Less than 1.00% above the threshold [0.0] [12:06:02] (03PS6) 10Addshore: docker: lintr image - linter for R [integration/config] - 10https://gerrit.wikimedia.org/r/378831 (https://phabricator.wikimedia.org/T176194) [12:06:13] !log docker push docker.io/wmfreleng/ & latest (PS6 of https://gerrit.wikimedia.org/r/378831 ) [12:06:17] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [12:07:12] (03PS4) 10Addshore: Add experimental lintr job for analytics/wmde/WDCM [integration/config] - 10https://gerrit.wikimedia.org/r/379818 (https://phabricator.wikimedia.org/T176194) [12:11:17] (03PS5) 10Addshore: Add experimental lintr job for analytics/wmde/WDCM [integration/config] - 10https://gerrit.wikimedia.org/r/379818 (https://phabricator.wikimedia.org/T176194) [12:16:32] (03CR) 10GoranSMilovanovic: [C: 031] docker: lintr image - linter for R [integration/config] - 10https://gerrit.wikimedia.org/r/378831 (https://phabricator.wikimedia.org/T176194) (owner: 10Addshore) [12:17:28] (03CR) 10GoranSMilovanovic: [C: 031] Add experimental lintr job for analytics/wmde/WDCM [integration/config] - 10https://gerrit.wikimedia.org/r/379818 (https://phabricator.wikimedia.org/T176194) (owner: 10Addshore) [12:20:06] (03CR) 10Addshore: [V: 031] "Passing when testing checking a repo with no R code in (so no lint failures)" [integration/config] - 10https://gerrit.wikimedia.org/r/379818 (https://phabricator.wikimedia.org/T176194) (owner: 10Addshore) [12:20:11] (03CR) 10Addshore: [C: 032] docker: lintr image - linter for R [integration/config] - 10https://gerrit.wikimedia.org/r/378831 (https://phabricator.wikimedia.org/T176194) (owner: 10Addshore) [12:20:16] woo! [12:20:27] (03CR) 10Addshore: [V: 031 C: 032] Add experimental lintr job for analytics/wmde/WDCM [integration/config] - 10https://gerrit.wikimedia.org/r/379818 (https://phabricator.wikimedia.org/T176194) (owner: 10Addshore) [12:20:28] addshore: I guess just always copy/archive the files? [12:20:39] some publishers have options such as "only run if build passed" [12:21:50] (03Merged) 10jenkins-bot: docker: lintr image - linter for R [integration/config] - 10https://gerrit.wikimedia.org/r/378831 (https://phabricator.wikimedia.org/T176194) (owner: 10Addshore) [12:24:30] (03PS6) 10Addshore: Add experimental lintr job for analytics/wmde/WDCM [integration/config] - 10https://gerrit.wikimedia.org/r/379818 (https://phabricator.wikimedia.org/T176194) [12:24:34] (03CR) 10Addshore: [C: 032] Add experimental lintr job for analytics/wmde/WDCM [integration/config] - 10https://gerrit.wikimedia.org/r/379818 (https://phabricator.wikimedia.org/T176194) (owner: 10Addshore) [12:25:44] (03Merged) 10jenkins-bot: Add experimental lintr job for analytics/wmde/WDCM [integration/config] - 10https://gerrit.wikimedia.org/r/379818 (https://phabricator.wikimedia.org/T176194) (owner: 10Addshore) [12:25:49] (03CR) 10Addshore: [C: 032] fabric: Add command to pull docker image on all hosts [integration/config] - 10https://gerrit.wikimedia.org/r/378783 (owner: 10Addshore) [12:25:54] (03PS6) 10Addshore: fabric: Add command to pull docker image on all hosts [integration/config] - 10https://gerrit.wikimedia.org/r/378783 [12:25:57] (03CR) 10Addshore: [C: 032] fabric: Add command to pull docker image on all hosts [integration/config] - 10https://gerrit.wikimedia.org/r/378783 (owner: 10Addshore) [12:26:33] !log Reloading Zuul to deploy https://gerrit.wikimedia.org/r/379818 [12:26:37] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [12:26:56] (03Merged) 10jenkins-bot: fabric: Add command to pull docker image on all hosts [integration/config] - 10https://gerrit.wikimedia.org/r/378783 (owner: 10Addshore) [12:28:20] !log fab docker_pull_image:wmfreleng/lintr:v2017.09.26.12.04 [12:28:24] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [12:28:34] PROBLEM - Puppet errors on deployment-mediawiki06 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [12:31:04] addshore: I guess it is compiling the packages on each build isn't it ? [12:34:04] hashar: nope [12:34:16] as it is just a linter :) [12:34:49] right, now to make it a job that triggers on patch submit but non voting for the WDCM repo! [12:35:02] It's been quite nice building a job from the ground up [12:38:36] RECOVERY - Puppet errors on deployment-mediawiki06 is OK: OK: Less than 1.00% above the threshold [0.0] [12:44:01] (03PS6) 10Addshore: Low prio queue for libraryupdater [integration/config] - 10https://gerrit.wikimedia.org/r/380307 [12:44:04] (03CR) 10Addshore: [C: 032] Low prio queue for libraryupdater [integration/config] - 10https://gerrit.wikimedia.org/r/380307 (owner: 10Addshore) [12:45:06] (03Merged) 10jenkins-bot: Low prio queue for libraryupdater [integration/config] - 10https://gerrit.wikimedia.org/r/380307 (owner: 10Addshore) [12:45:32] !log Reloading Zuul to deploy - Low prio queue for libraryupdater [integration/config] - https://gerrit.wikimedia.org/r/380307 [12:45:36] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [12:52:40] addshore: have you reused the docker-run jjb macros for that ? [12:52:54] I wanna try to create a job for tox [12:52:55] no, as I'm going to change those macros in a sec :P [12:53:22] hmm, the lowprio queue thing didnt seem to work in my test. but maybe that because im on the whitelist for the main "test" pipeline still [12:54:11] does the project has jobs attached to a test-lowprio pipeline? [12:57:11] hashar: oh, you have to configure that per job D: [12:57:40] (03PS1) 10Addshore: docker: use macros in lintr-docker [integration/config] - 10https://gerrit.wikimedia.org/r/380736 [13:01:19] (03PS1) 10Aude: Update Wikidata branch [tools/release] - 10https://gerrit.wikimedia.org/r/380738 [13:06:41] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban), 10DNS, 10Operations, and 2 others: CI: operations-dns-lint broken due to missing Maxmind DB file - https://phabricator.wikimedia.org/T175864#3635184 (10hashar) 05Open>03Resolved a:03hashar Apparently that was transient or p... [13:07:15] !log adding deployment-videoscaler01 to deployment-prep (stretch-based video scaler) [13:07:19] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [13:09:47] (03PS2) 10Addshore: docker: use macros in lintr-docker [integration/config] - 10https://gerrit.wikimedia.org/r/380736 [13:11:01] RECOVERY - Puppet errors on deployment-mediawiki07 is OK: OK: Less than 1.00% above the threshold [0.0] [13:18:57] 10Continuous-Integration-Infrastructure (shipyard): Some docker slave still have old containers using old images - https://phabricator.wikimedia.org/T176623#3635213 (10Addshore) When jobs are killed due to log execution time such as https://integration.wikimedia.org/ci/job/lintr-docker/24/console this can happen... [13:26:14] 10Release-Engineering-Team, 10MediaWiki-Configuration, 10Patch-For-Review, 10Services (watching): Automate WMF wiki creation - https://phabricator.wikimedia.org/T158730#3635240 (10Gilles) [13:33:56] addshore: so zuul listen for events from Gerrit. Then it pass the event through each of the pipeline [13:34:00] 10Continuous-Integration-Infrastructure (shipyard): Some docker slave still have old containers using old images - https://phabricator.wikimedia.org/T176623#3635269 (10Addshore) It actually looks like these containers continue to run and use resources on the node after jenkins has killed the process that started... [13:34:35] addshore: if test rejects the library updater (based on the email being rejected) zuul is not going to trigger anything for that pipeline [13:34:42] so there is a bunch of copy paste to do everywhere :( [13:35:13] hashar: I was thinking, it might be nice to have a script generate the jjb file for mediawiki-extensions [13:35:25] and then just have some sort of matrix of all the extensions and what jobs should be run for them [13:35:30] another solution would be to blacklist it from the test pipeline (and remove the lowprio one) [13:35:44] then have the updater to +2 the patch whenever gate-and-submit has less than eg 4 changes [13:36:03] that might be a nice option [13:36:25] last time I did a mass change [13:36:31] (03CR) 10Aude: [C: 032] Update Wikidata branch [tools/release] - 10https://gerrit.wikimedia.org/r/380738 (owner: 10Aude) [13:36:41] i sent a patch via the test pipeline every 120 seconds or something like that [13:37:20] then when tests passed, I have git push them [13:37:32] (03CR) 10Addshore: [C: 032] docker: use macros in lintr-docker [integration/config] - 10https://gerrit.wikimedia.org/r/380736 (owner: 10Addshore) [13:37:46] and for some other mass changes that had zero impact on tests, I just git push directly to gerrit -bypassing ci- [13:39:16] (03Merged) 10jenkins-bot: Update Wikidata branch [tools/release] - 10https://gerrit.wikimedia.org/r/380738 (owner: 10Aude) [13:41:26] (03Merged) 10jenkins-bot: docker: use macros in lintr-docker [integration/config] - 10https://gerrit.wikimedia.org/r/380736 (owner: 10Addshore) [13:42:53] 10Continuous-Integration-Infrastructure (shipyard): When jenkins kills a build due to max execution time the docker containers stay running - https://phabricator.wikimedia.org/T176747#3635359 (10Addshore) [13:43:01] hashar: ^^ thats an interesting one [13:43:22] 10Continuous-Integration-Infrastructure (shipyard): When jenkins kills a build due to max execution time the docker containers stay running - https://phabricator.wikimedia.org/T176747#3635373 (10Addshore) [13:43:54] 10Continuous-Integration-Infrastructure (shipyard): Some docker slave still have old containers using old images - https://phabricator.wikimedia.org/T176623#3631625 (10Addshore) I have split the last 2 comments into T176747 as container continuing to run is a different issue to old containers not running but han... [13:46:14] Project selenium-VisualEditor » firefox,beta,Linux,BrowserTests build #534: 04FAILURE in 2 min 13 sec: https://integration.wikimedia.org/ci/job/selenium-VisualEditor/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=BrowserTests/534/ [13:52:49] (03PS1) 10Addshore: Add lintr-docker-non-voting [integration/config] - 10https://gerrit.wikimedia.org/r/380746 [13:54:41] (03CR) 10jerkins-bot: [V: 04-1] Add lintr-docker-non-voting [integration/config] - 10https://gerrit.wikimedia.org/r/380746 (owner: 10Addshore) [13:57:44] (03PS2) 10Addshore: Add lintr-docker-non-voting [integration/config] - 10https://gerrit.wikimedia.org/r/380746 [13:57:59] (03PS3) 10Addshore: Add lintr-docker-non-voting [integration/config] - 10https://gerrit.wikimedia.org/r/380746 (https://phabricator.wikimedia.org/T176194) [13:58:58] !log added hashar to https://hub.docker.com/u/wmfreleng [13:59:02] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [14:00:53] 10Release-Engineering-Team (Watching / External), 10Contributors-Team, 10MobileFrontend, 10Operations, and 2 others: Diff page produces 503 on first visit - https://phabricator.wikimedia.org/T176637#3632060 (10phuedx) @Jdlrobson: Was/is this specific to the Beta Cluster? I can't reproduce this in productio... [14:02:20] 10Continuous-Integration-Infrastructure (shipyard), 10WMDE-Analytics-Engineering, 10Patch-For-Review, 10User-Addshore: Have CI run lintr for analytics/wmde/WDCM R files - https://phabricator.wikimedia.org/T176194#3635446 (10Addshore) So, as an experimental test everything is working. The patch above adds i... [14:02:21] PROBLEM - Puppet errors on deployment-videoscaler01 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [14:02:50] 10Continuous-Integration-Infrastructure (shipyard), 10WMDE-Analytics-Engineering, 10Patch-For-Review, 10User-Addshore, 10WMDE-QWERTY-Sprint-2017-09-19: Have CI run lintr for analytics/wmde/WDCM R files - https://phabricator.wikimedia.org/T176194#3635447 (10Addshore) [14:09:42] 10Release-Engineering-Team (Watching / External), 10Contributors-Team, 10MobileFrontend, 10Operations, 10Readers-Web-Backlog (Tracking): Diff page produces 503 on beta cluster on first visit - https://phabricator.wikimedia.org/T176637#3635473 (10Jdlrobson) [14:10:22] 10Release-Engineering-Team (Watching / External), 10Contributors-Team, 10MobileFrontend, 10Operations, 10Readers-Web-Backlog (Tracking): Diff page consistently produces 503 on beta cluster on first visit - https://phabricator.wikimedia.org/T176637#3632060 (10Jdlrobson) [14:17:23] RECOVERY - Puppet errors on deployment-videoscaler01 is OK: OK: Less than 1.00% above the threshold [0.0] [14:33:58] Project selenium-WikiLove » firefox,beta,Linux,BrowserTests build #528: 04FAILURE in 1 min 58 sec: https://integration.wikimedia.org/ci/job/selenium-WikiLove/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=BrowserTests/528/ [15:12:59] 10Release-Engineering-Team (Kanban), 10Wikimedia-Blog-Content: Code Health and the Code Health Group - https://phabricator.wikimedia.org/T175187#3585511 (10MelodyKramer) Draft is submitted/edited, waiting for approval. [15:25:38] Yippee, build fixed! [15:25:38] Project selenium-MobileFrontend » firefox,beta,Linux,BrowserTests build #575: 09FIXED in 3 min 37 sec: https://integration.wikimedia.org/ci/job/selenium-MobileFrontend/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=BrowserTests/575/ [15:30:18] (03PS5) 10Hashar: dockerfiles: config file + http_proxy support [integration/config] - 10https://gerrit.wikimedia.org/r/379507 [15:30:33] and I rebased the http_proxy support ^ :D [15:30:34] off! [15:41:41] (03CR) 10Addshore: "It looks like building with this patch (when not changing anything / setting anything) means images aren't using the build cache :(" [integration/config] - 10https://gerrit.wikimedia.org/r/379507 (owner: 10Hashar) [15:46:42] (03PS5) 10Addshore: Migrate 'mediawiki-core-phpcs' job to docker [integration/config] - 10https://gerrit.wikimedia.org/r/379479 (owner: 10Legoktm) [15:46:49] (03CR) 10jerkins-bot: [V: 04-1] Migrate 'mediawiki-core-phpcs' job to docker [integration/config] - 10https://gerrit.wikimedia.org/r/379479 (owner: 10Legoktm) [15:50:16] (03PS6) 10Addshore: Experimental Migrate 'mediawiki-core-phpcs' job to docker [integration/config] - 10https://gerrit.wikimedia.org/r/379479 (owner: 10Legoktm) [15:50:22] (03CR) 10jerkins-bot: [V: 04-1] Experimental Migrate 'mediawiki-core-phpcs' job to docker [integration/config] - 10https://gerrit.wikimedia.org/r/379479 (owner: 10Legoktm) [15:50:24] (03PS7) 10Addshore: Experimental Migrate 'mediawiki-core-phpcs' job to docker [integration/config] - 10https://gerrit.wikimedia.org/r/379479 (owner: 10Legoktm) [15:51:44] (03CR) 10jerkins-bot: [V: 04-1] Experimental Migrate 'mediawiki-core-phpcs' job to docker [integration/config] - 10https://gerrit.wikimedia.org/r/379479 (owner: 10Legoktm) [15:53:45] (03PS8) 10Addshore: Experimental Migrate 'mediawiki-core-phpcs' job to docker [integration/config] - 10https://gerrit.wikimedia.org/r/379479 (owner: 10Legoktm) [15:54:57] (03CR) 10Addshore: "So, right now this doesn't actually get the code that it needs to test anywhere... :D" [integration/config] - 10https://gerrit.wikimedia.org/r/379479 (owner: 10Legoktm) [15:58:15] (03PS9) 10Addshore: Experimental Migrate 'mediawiki-core-phpcs' job to docker [integration/config] - 10https://gerrit.wikimedia.org/r/379479 (owner: 10Legoktm) [15:58:59] (03CR) 10Addshore: "PS9 adds a zuul-cloner run on the jenkins slave itself." [integration/config] - 10https://gerrit.wikimedia.org/r/379479 (owner: 10Legoktm) [15:59:10] (03CR) 10jerkins-bot: [V: 04-1] Experimental Migrate 'mediawiki-core-phpcs' job to docker [integration/config] - 10https://gerrit.wikimedia.org/r/379479 (owner: 10Legoktm) [15:59:45] (03PS10) 10Addshore: Experimental Migrate 'mediawiki-core-phpcs' job to docker [integration/config] - 10https://gerrit.wikimedia.org/r/379479 (owner: 10Legoktm) [16:01:01] !log docker push docker.io/wmfreleng/mediawiki-phpcs:v2017.09.26.15.45 & latest (From PS10 of https://gerrit.wikimedia.org/r/379479) [16:01:05] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [16:01:56] (03CR) 10Addshore: "image pushed and https://integration.wikimedia.org/ci/job/mediawiki-core-phpcs-docker/ added to jenkins" [integration/config] - 10https://gerrit.wikimedia.org/r/379479 (owner: 10Legoktm) [16:14:44] (03PS11) 10Addshore: Experimental Migrate 'mediawiki-core-phpcs' job to docker [integration/config] - 10https://gerrit.wikimedia.org/r/379479 (owner: 10Legoktm) [16:15:02] !log docker push docker.io/wmfreleng/zuul-cloner:v2017.09.26.16.09 & latest (from PS11 of https://gerrit.wikimedia.org/r/379479) [16:15:06] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [16:18:01] (03CR) 10Addshore: "A full run as in PS11 can be seen @ https://integration.wikimedia.org/ci/job/mediawiki-core-phpcs-docker/4/consoleFull" (033 comments) [integration/config] - 10https://gerrit.wikimedia.org/r/379479 (owner: 10Legoktm) [16:18:09] legoktm: ^^ for when you wake up [16:18:33] 10Gitblit-Deprecate, 10Cleanup, 10Project-Admins: Archive #Gitblit-Deprecate - https://phabricator.wikimedia.org/T138986#3636039 (10Dzahn) 12:14 < andrewbogott> !log gitblit rebooting 'test' instance as it is unreachable ^ Is this task (and subtasks) still being worked on by anyone? Since the labs instance... [16:21:14] 10Gitblit-Deprecate, 10Cleanup, 10Project-Admins: Archive #Gitblit-Deprecate - https://phabricator.wikimedia.org/T138986#3636046 (10Paladox) I think the gitblit migration is complete as there has been low complaints and it has been over a year. I think we should archive the gitblit project. [16:22:12] 10Gitblit-Deprecate, 10Cleanup, 10Project-Admins: Archive #Gitblit-Deprecate - https://phabricator.wikimedia.org/T138986#3636052 (10TerraCodes) {T141965} still needs to be dealt with first. [16:22:57] 10Gitblit-Deprecate, 10Cleanup, 10Project-Admins: Archive #Gitblit-Deprecate - https://phabricator.wikimedia.org/T138986#3636054 (10Dzahn) I agree, i think about 95% of the old URLs have been covered with redirects and this was about the remaining 5% or something like that but it's good enough as it is and i... [16:25:40] 10Gitblit-Deprecate, 10Diffusion: Redirect git.wikimedia.org HEAD URLs to Diffusion - https://phabricator.wikimedia.org/T141965#3636071 (10greg) >>! In T141965#3192601, @Krinkle wrote: > It seems Diffusion breaks when trying to "Browse tags" on a repo that doesn't yet have tags. > > For example, (03CR) 10Addshore: Experimental Migrate 'mediawiki-core-phpcs' job to docker (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/379479 (owner: 10Legoktm) [16:58:54] (03CR) 10Hashar: dockerfiles: config file + http_proxy support (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/379507 (owner: 10Hashar) [16:59:03] (03PS6) 10Hashar: dockerfiles: config file + http_proxy support [integration/config] - 10https://gerrit.wikimedia.org/r/379507 [17:00:31] (03CR) 10Hashar: "https://gerrit.wikimedia.org/r/#/c/379507/5..6/dockerfiles/build.py would fix the issue. I guess the build arg was always injected and th" [integration/config] - 10https://gerrit.wikimedia.org/r/379507 (owner: 10Hashar) [17:01:31] (03PS1) 10Hashar: dockerfiles: sort list of images candidates [integration/config] - 10https://gerrit.wikimedia.org/r/380787 [17:03:25] (03CR) 10Addshore: [C: 032] dockerfiles: sort list of images candidates [integration/config] - 10https://gerrit.wikimedia.org/r/380787 (owner: 10Hashar) [17:05:12] (03Merged) 10jenkins-bot: dockerfiles: sort list of images candidates [integration/config] - 10https://gerrit.wikimedia.org/r/380787 (owner: 10Hashar) [17:07:04] (03CR) 10Addshore: dockerfiles: config file + http_proxy support (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/379507 (owner: 10Hashar) [17:08:17] PROBLEM - Puppet errors on integration-slave-docker-1001 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [17:13:15] (03PS1) 10Umherirrender: Add BlueSpiceExtensions as dependencies [integration/config] - 10https://gerrit.wikimedia.org/r/380790 [17:14:22] (03CR) 10Umherirrender: "See https://gerrit.wikimedia.org/r/#/c/376789/ for a failing test" [integration/config] - 10https://gerrit.wikimedia.org/r/380790 (owner: 10Umherirrender) [17:18:22] addshore: yaaaaay [17:18:38] legoktm: yeh [17:18:48] I need to ask hashar how the git cache works [17:18:58] and also, figure out if we want to point to gerrit or something else? [17:19:09] and if we can only install mediawiki code sniffer and not everything else [17:19:11] Can we do something like https://github.com/wikimedia/labs-libraryupgrader/blob/master/Dockerfile to prepopulate composer cache? [17:19:45] (Also my laptop is dead so I don't have access to gerrit right now) [17:19:46] Yeh, I was thinking we could actually have an image which is literally mediawiki-composer-cache, that other images can just copy the cache out of? :P [17:19:57] Sure [17:20:09] Just ~/.composer I think? [17:21:40] (03PS2) 10Dduvall: WIP Service pipeline DSL [integration/config] - 10https://gerrit.wikimedia.org/r/380551 (https://phabricator.wikimedia.org/T175297) [17:22:24] 10Release-Engineering-Team (Kanban), 10Release Pipeline, 10Patch-For-Review: Define new Jenkins pipeline for container build phase - https://phabricator.wikimedia.org/T175297#3636225 (10dduvall) p:05Triage>03Normal a:03dduvall [17:24:41] greg-g: https://phabricator.wikimedia.org/T176637 could potentially be an issue with the new wikidiff2 version that's on beta [17:40:01] 10Release-Engineering-Team (Kanban), 10Release Pipeline, 10Patch-For-Review: Define new Jenkins pipeline for container build phase - https://phabricator.wikimedia.org/T175297#3636244 (10dduvall) An [[ https://integration.wikimedia.org/ci/job/service-pipeline | experimental job ]] has been created from the cu... [17:43:20] RECOVERY - Puppet errors on integration-slave-docker-1001 is OK: OK: Less than 1.00% above the threshold [0.0] [18:02:50] (03PS2) 10Umherirrender: Update test config for BlueSpice extensions [integration/config] - 10https://gerrit.wikimedia.org/r/380790 [18:07:27] 10Gerrit, 10Release-Engineering-Team (Backlog), 10Operations, 10Patch-For-Review: Reimage gerrit2001 as stretch - https://phabricator.wikimedia.org/T168562#3636290 (10Paladox) I guess we can close this as resolved now? [18:08:04] 10Release-Engineering-Team (Kanban), 10Release Pipeline, 10Patch-For-Review: Define new Jenkins pipeline for container build phase - https://phabricator.wikimedia.org/T175297#3636303 (10dduvall) The more specifically relevant bug may be [[ https://issues.jenkins-ci.org/browse/JENKINS-44609 | this one ]] whic... [18:08:35] 10Gerrit, 10Release-Engineering-Team, 10Operations: Reimage cobalt as stretch - https://phabricator.wikimedia.org/T176774#3636304 (10Paladox) [18:11:01] 10Gerrit, 10Release-Engineering-Team (Backlog), 10Scap, 10Patch-For-Review: Deploy gerrit with scap3 - https://phabricator.wikimedia.org/T157414#3636324 (10Paladox) We only need https://gerrit.wikimedia.org/r/#/c/378768/ and https://gerrit.wikimedia.org/r/#/c/379136/ merged to close this task as resolved a... [18:13:41] (03CR) 10Hashar: dockerfiles: config file + http_proxy support (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/379507 (owner: 10Hashar) [18:13:48] (03PS7) 10Hashar: dockerfiles: config file + http_proxy support [integration/config] - 10https://gerrit.wikimedia.org/r/379507 [18:16:20] (03CR) 10Hashar: "When setting http_proxy, the lintr image fails to build because it tries to use the http_proxy value:" [integration/config] - 10https://gerrit.wikimedia.org/r/379507 (owner: 10Hashar) [18:24:11] 10Gerrit, 10Release-Engineering-Team, 10Operations: Reimage cobalt as stretch - https://phabricator.wikimedia.org/T176774#3636363 (10Dzahn) Not wrong, just too early for that. gerrit2001 isn't done yet, it's in the middle of the process. Yes to the renaming part... later. [18:24:29] 10Gerrit, 10Release-Engineering-Team, 10Operations: Reimage cobalt as stretch - https://phabricator.wikimedia.org/T176774#3636364 (10Dzahn) p:05Triage>03Lowest [18:30:58] 10Gerrit, 10Release-Engineering-Team (Backlog), 10Operations, 10Patch-For-Review: Reimage gerrit2001 as stretch - https://phabricator.wikimedia.org/T168562#3636377 (10Dzahn) Depends how you define it. If it's only about OS installation and applying the puppet roles without errors, yes. But Gerrit the serv... [18:32:16] (03PS3) 10Umherirrender: Update test config [integration/config] - 10https://gerrit.wikimedia.org/r/380790 [18:35:21] (03PS3) 10Dduvall: WIP Service pipeline DSL [integration/config] - 10https://gerrit.wikimedia.org/r/380551 (https://phabricator.wikimedia.org/T175297) [18:51:52] (03PS4) 10Dduvall: WIP Service pipeline DSL [integration/config] - 10https://gerrit.wikimedia.org/r/380551 (https://phabricator.wikimedia.org/T175297) [18:54:26] (03PS8) 10Hashar: dockerfiles: config file + http_proxy support [integration/config] - 10https://gerrit.wikimedia.org/r/379507 [19:00:06] 10Release-Engineering-Team (Kanban), 10Release Pipeline, 10Patch-For-Review: Define new Jenkins pipeline for container build phase - https://phabricator.wikimedia.org/T175297#3636469 (10dduvall) [[ https://gerrit.wikimedia.org/r/#/c/380551/4 | Patchset 4]] successfully builds the production image but [[ http... [19:30:24] 10Gerrit, 10Release-Engineering-Team (Backlog), 10Operations, 10Patch-For-Review: Reimage gerrit2001 as stretch - https://phabricator.wikimedia.org/T168562#3636612 (10Dzahn) [19:30:26] 10Gerrit, 10Release-Engineering-Team (Next), 10Operations: Gerrit is failing to start gerrit-ssh on gerrit2001 - https://phabricator.wikimedia.org/T176532#3636611 (10Dzahn) [19:37:27] 10Gerrit, 10Release-Engineering-Team (Next), 10Operations: Gerrit is failing to start gerrit-ssh on gerrit2001 - https://phabricator.wikimedia.org/T176532#3636684 (10Paladox) I think we found the likly culprit (Chad) found this P6046. It's not connecting to the db due to a firewall preventing it from being a... [19:38:22] 10Release-Engineering-Team (Watching / External), 10Contributors-Team, 10MobileFrontend, 10Operations, and 2 others: Diff page consistently produces 503 on beta cluster on first visit - https://phabricator.wikimedia.org/T176637#3636688 (10greg) ``` 17:24 < legoktm> greg-g: https://phabricator.wikimedia.o... [19:39:26] greg-g: the people responsible for wikidiff2 today are MaxSem and wmde tcb team [19:39:39] 10Gerrit, 10Release-Engineering-Team (Next), 10Operations: Gerrit is failing to start gerrit-ssh on gerrit2001 - https://phabricator.wikimedia.org/T176532#3636695 (10Dzahn) Here is a change to add firewall rules to mariadb::misc to fix that. https://gerrit.wikimedia.org/r/#/c/380827/ [19:40:02] legoktm: ty [19:41:19] hrm [19:45:52] 10Release-Engineering-Team (Watching / External), 10Contributors-Team, 10MobileFrontend, 10Operations, and 2 others: Diff page consistently produces 503 on beta cluster on first visit - https://phabricator.wikimedia.org/T176637#3636703 (10greg) Adding @maxsem and #tcb-team per Lego on IRC [19:46:11] let's see, who else can I add? :) [19:46:49] Heh [19:47:02] This irc2phab bridge is working pretty well ;) [19:47:04] spam everybody [19:47:10] legoktm: I aim to please [19:47:23] MaxSem: almost there :P [19:50:31] remind me what's the betalabs' analog of mwlog1001? [19:50:58] It might still be called deployment-fluorine? [19:51:45] nope [19:51:45] There's also logstash-beta.wmflabs iirc [19:51:55] tin and mwlog are two different things:) [19:52:33] I said fluorine? [19:52:48] duh [19:52:50] :P [19:53:38] https://tools.wmflabs.org/openstack-browser/server/deployment-fluorine02.deployment-prep.eqiad.wmflabs [19:53:44] That looks like it [19:54:56] thanks [20:00:14] 10Release-Engineering-Team, 10Cleanup, 10Repository-Admins, 10User-MarcoAurelio: Archive Extension:Phalanx - https://phabricator.wikimedia.org/T176665#3633249 (10greg) (Just to be clear, the github stuff doesn't need someone in acl-releng, the group of people who are admins/owners of the Wikimedia github "... [20:02:24] 10Release-Engineering-Team (Watching / External), 10Contributors-Team, 10MobileFrontend, 10Operations, and 2 others: Diff page consistently produces 503 on beta cluster on first visit - https://phabricator.wikimedia.org/T176637#3636823 (10MaxSem) Stacktrace: {F9833532} [20:05:30] also, now a diff displays for me [20:09:09] greg-g: only acl*releng has access to K13, that's why I added the tag [20:09:25] also https://en.wikipedia.org/w/index.php?title=Special:ListUsers&group=monitor <-- this needs SQL cleanup [20:13:02] 10Release-Engineering-Team, 10Cleanup, 10Repository-Admins, 10User-MarcoAurelio: Archive Extension:Phalanx - https://phabricator.wikimedia.org/T176665#3636902 (10MarcoAurelio) @greg I added the tag based on {K13}'s policies. But if you can point me to the appropriate phabricator project to ping on sucesive... [20:13:26] tabbycat: no action will be taken with that by just mentioning here :) [20:13:51] 10Release-Engineering-Team, 10Cleanup, 10Repository-Admins, 10User-MarcoAurelio: Archive Extension:Phalanx - https://phabricator.wikimedia.org/T176665#3636918 (10MarcoAurelio) [20:14:08] 10Release-Engineering-Team, 10Cleanup, 10Repository-Admins, 10User-MarcoAurelio: Archive Extension:Phalanx - https://phabricator.wikimedia.org/T176665#3633249 (10MarcoAurelio) 05Open>03Resolved [20:15:48] 10Release-Engineering-Team, 10Cleanup, 10Repository-Admins, 10User-MarcoAurelio: Archive Extension:Phalanx - https://phabricator.wikimedia.org/T176665#3636922 (10greg) K13 is the password for the syncing, not for the wikimedia organization in github (who has access to do these actions). Here's the list: h... [20:16:24] (03PS4) 10Umherirrender: Update test config [integration/config] - 10https://gerrit.wikimedia.org/r/380790 [20:17:12] greg-g: T176798 :) [20:17:13] T176798: Remove 'monitor' group from enwiki - https://phabricator.wikimedia.org/T176798 [20:17:49] (03PS5) 10Umherirrender: Update test config [integration/config] - 10https://gerrit.wikimedia.org/r/380790 [20:18:23] (03CR) 10Umherirrender: "Now all necessary tests are added." [integration/config] - 10https://gerrit.wikimedia.org/r/380790 (owner: 10Umherirrender) [20:24:32] 10Release-Engineering-Team, 10Cleanup, 10Repository-Admins, 10User-MarcoAurelio: Archive Extension:Phalanx - https://phabricator.wikimedia.org/T176665#3637019 (10MarcoAurelio) >>! In T176665#3636922, @greg wrote: > K13 is the password for the syncing, not for the wikimedia organization in github (who has a... [20:27:38] no_justification: Did you make the VE wmf.1 branch creation commit by hand or is the script broken? [20:27:39] no_justification: https://phabricator.wikimedia.org/rEVEDf526a52a82188494b4d173fa154fa9df5bda3eac [20:28:12] Ok wtf. [20:28:16] Note that the Committer is you but the Author is Ed, and the diff is https://gerrit.wikimedia.org/r/#/c/380731/ which Gerrit believes is not in wmf.1, but it was merged in time and made it [20:28:26] So my theory is you --amend-ed the commit message away [20:28:45] I don't do anything like that [20:28:46] At all [20:28:54] Well it's got your name on it :P [20:31:16] In this particular case, that one commit broke VE, and a fix was committed quickly after, but the fix wasn't cherry-picked because Gerrit said that the broken commit had missed the train [20:31:25] Something is up. [20:31:40] Then I found the breakage in production on mw.org, cherry-picked the fix, and James said "wtf didn't that commit miss the train" [20:31:56] (03CR) 10jerkins-bot: [V: 04-1] Update test config [integration/config] - 10https://gerrit.wikimedia.org/r/380790 (owner: 10Umherirrender) [20:31:56] I guess the script is odd. [20:32:14] 'Cos there's no way no_justification would type "--amend" without noticing. [20:32:23] https://phabricator.wikimedia.org/T175324 [20:32:27] Related to ^ likely [20:32:28] James_F: Didn't we see this two weeks ago? Where you said you were the author of the branch creation commit for VE? [20:32:32] Which is a rabbit hole we can't figure [20:32:42] thcipriani: Blagh [20:32:51] Oh. [20:33:21] ugh [20:33:29] Yes. [20:34:08] https://phabricator.wikimedia.org/rEVEDda0acffe7203daef58da1afb42130eac94034554 for wmf.17 is like this, as is https://phabricator.wikimedia.org/rEVED3a991344156f9eb4d43719f67c7480069960b7d4 for .18 and https://phabricator.wikimedia.org/rEVEDe437600324938088fc27824fac6cf9b52edba5df for .19 [20:34:32] And wmf.16: https://phabricator.wikimedia.org/rEVED10b03de031719bae0ec3ae762e25e864e7222fb8 [20:34:48] We shouldn't even be /making/ commits like that anymore for branches. [20:34:52] Earlier branches seem to have been killed, so I can't check if this has always been broken and we've just not noticed. [20:36:06] Shit. We do do this [20:36:43] Why wouldn't we make commits like that? We have to update .gitreview [20:37:13] We don't update .gitreview except on master [20:37:20] We haven't for well over a year or so [20:37:40] Hah, you're right, it doesn't update .gitreview [20:37:51] The branch is no longer in there [20:38:10] Which is why it's always a nightmare to do production back-ports nowadays. [20:38:26] Does git-review not correct for this automatically? [20:38:40] I guess people don't manually submit cherry-picks that often any more [20:38:55] Pretty much only me, I imagine. No-one else has to deal with a submodule. [20:39:07] don't use git-review [20:39:20] I did one recently because I needed to cherry-pick a stack of two commits [20:39:28] Gerrit isn't smart enough to rebase the second one correctly [20:39:39] And git-review worked for me then, I'm pretty sure [20:39:56] But perhaps that was because of the dependency [20:40:27] In any case, no_justification is right that Flow does not have a "Creating branch" commit in wmf1. [20:40:45] no_justification: Eurgh. [20:40:54] I'm digging through the blame. [20:40:56] Core needs one because of the submodules, but VE doesn't need one /even though/ it has submodules [20:41:03] Sure. [20:41:06] Because VE manages its submodule slightly differently [20:41:44] well for extensions, not modifying .gitreview coupled with: https://github.com/wikimedia/mediawiki-tools-release/blob/master/make-wmf-branch/MakeWmfBranch.php#L194 likely explains those commits [20:42:25] I would guess that's been happening since the elimination of the modification of the .gitreview file (which was causing different issues at the time) [20:42:44] Somewhere around there. [20:43:21] modifying the sha1 of the commit on the tip of each branched submodule [20:43:23] Why does that --amend ? [20:43:32] That's a very very good question [20:43:38] It's been there since...a long time [20:43:44] At least back through 2014 [20:43:53] Was it counting on a more rudimentary commit having been created already? [20:44:04] Probably, from Fixup gitreview [20:44:16] But either way, amending is *not* what we wanted here [20:45:15] (03CR) 10Umherirrender: "recheck" [integration/config] - 10https://gerrit.wikimedia.org/r/380790 (owner: 10Umherirrender) [20:49:34] chad@notsexy /a/release/make-wmf-branch (master)$ git log --oneline --grep=gitreview . [20:49:34] 6d559ed Stop mangling .gitreview [20:49:34] 568f822 Merge "Ensure changes to an extension's .gitreview file" [20:49:35] 9d3ad22 Ensure changes to an extension's .gitreview file [20:49:35] a35428b Checkout new branch so we get fixed .gitreview in right branch [20:49:35] 49c4d61 Fix .gitreview in extension branches when we branch for WMF version [20:49:35] ca92892 Update .gitreview and version number before doing patches [20:49:35] c3a8b41 Apply patches before poking version and .gitreview file [20:49:35] 31368b2 Automatically update .gitreview file default branch, and hten it can be commited in "Applied patches to..." [20:50:02] So it was definitely *not* amending in 31368b2. And by the time 6d559ed came around, it just exposed the subtle issue. [20:50:08] Somewhere in between it broke [20:50:23] (03CR) 10jerkins-bot: [V: 04-1] Update test config [integration/config] - 10https://gerrit.wikimedia.org/r/380790 (owner: 10Umherirrender) [20:50:36] Curious why we haven't noticed it before. Cleaning up old branches is kinda a new-ish thing [20:50:46] Phab might make it more obvious probably [20:50:51] Since they don't go through gerrit [20:50:58] er +2 and such [21:13:11] I figured it out. [21:13:15] Commit incoming [21:13:20] (in a bit) [21:19:42] 10Release-Engineering-Team (Kanban), 10Technical-Debt: Setup Tech Debt SIG meetings - https://phabricator.wikimedia.org/T173351#3637308 (10Jrbranaa) Held both initially scheduled video-based SIG sessions. Notes available. Will look to setup an IRC based session too. [21:19:59] 10Release-Engineering-Team (Kanban), 10Technical-Debt: Setup Tech Debt SIG meetings - https://phabricator.wikimedia.org/T173351#3637312 (10Jrbranaa) 05Open>03Resolved [21:21:08] (03PS1) 10Chad: make-wmf-branch: Stop amending commits for branches with submodules [tools/release] - 10https://gerrit.wikimedia.org/r/380885 (https://phabricator.wikimedia.org/T175324) [21:22:18] PROBLEM - Puppet errors on aptly is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [21:27:13] 10Gerrit, 10Release-Engineering-Team (Next), 10Operations: Gerrit is failing to start gerrit-ssh on gerrit2001 - https://phabricator.wikimedia.org/T176532#3637335 (10Paladox) @Dazhn we are going to have to manually open the port per jynus for now [21:29:04] (03CR) 10Thcipriani: make-wmf-branch: Stop amending commits for branches with submodules (031 comment) [tools/release] - 10https://gerrit.wikimedia.org/r/380885 (https://phabricator.wikimedia.org/T175324) (owner: 10Chad) [21:32:33] (03CR) 10Hashar: [C: 032] Typo fix in docs [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/380637 (owner: 10MarcoAurelio) [21:33:30] (03Merged) 10jenkins-bot: Typo fix in docs [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/380637 (owner: 10MarcoAurelio) [21:34:27] codesniffer ftw [22:02:09] Jenkins seems to be broken - its complaining that oojs too old and composer update needs to be run [22:02:56] Also for some reason i cannot edit T176055 [22:02:57] T176055: Update of QueryPages failing on commons with "MASTER_POS_WAIT() or MASTER_GTID_WAIT() failed: MySQL server has gone away" - https://phabricator.wikimedia.org/T176055 [22:03:41] Project beta-code-update-eqiad build #174324: 04FAILURE in 40 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/174324/ [22:08:40] PROBLEM - Puppet errors on saucelabs-02 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [22:13:40] Yippee, build fixed! [22:13:41] Project beta-code-update-eqiad build #174325: 09FIXED in 40 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/174325/ [22:19:25] bawolff: re phab, is this what you see? https://phabricator.wikimedia.org/T176769 [22:20:02] Yes [22:20:47] It also has the weird bug keyword at the top ive never seen before [22:20:52] 10Release-Engineering-Team (Kanban), 10Phabricator, 10User-Matthewrbowker: "No edit forms" when attempting to edit task - https://phabricator.wikimedia.org/T176769#3637520 (10greg) p:05Triage>03Unbreak! a:03mmodell Yeah, that's bad. What are the options, @mmodell ? [22:21:24] bawolff: task types, see https://phabricator.wikimedia.org/T93499 (and a thread on the teampractices@ public mailing list) [22:21:31] 10Release-Engineering-Team (Kanban), 10Phabricator, 10User-Matthewrbowker: "No edit forms" when attempting to edit task - https://phabricator.wikimedia.org/T176769#3636126 (10Bawolff) Also happening for me on T176055 [22:22:26] 10Release-Engineering-Team (Kanban), 10Phabricator, 10User-Matthewrbowker: "No edit forms" when attempting to edit task - https://phabricator.wikimedia.org/T176769#3637529 (10Matthewrbowker) >>! In T176769#3637525, @Bawolff wrote: > Also happening for me on T176055 I am also encountering this issue on that... [22:23:57] 10Release-Engineering-Team (Kanban), 10Phabricator, 10User-Matthewrbowker: "No edit forms" when attempting to edit task - https://phabricator.wikimedia.org/T176769#3637530 (10greg) Yes, it has to do with the ACLs for the form that is associated with the BUG task type. [22:24:11] 10Release-Engineering-Team (Kanban), 10Phabricator, 10User-Matthewrbowker: "No edit forms" when attempting to edit task - https://phabricator.wikimedia.org/T176769#3637531 (10mmodell) 05Open>03Resolved Uhm, I locked down the form because people complained that the bug task type would be misused and there... [22:26:39] twentyafterfour: hmmmm, what should we do here? it seems there's the choice of disabling the access to the task type (bug) and closing/opening as non-bug-task-type any tasks already of the type, or making it so the bug task type is working as intended. [22:27:44] greg-g: it should currently be possible to edit those tasks but not submit new ones (if I've configured it correctly) - I didn't mean to lock down editing of existing tasks before [22:27:56] so I think this will be good for now [22:28:00] got it [22:28:04] thanks [22:28:27] someone could still create those tasks if they go to the form directly but it won't show up in menus anymore [22:30:04] good enough [22:30:51] Thanks i can edit now :) [22:32:16] 10Release-Engineering-Team (Kanban), 10User-greg: Fill in quarterly review slides for our stuff - https://phabricator.wikimedia.org/T176820#3637540 (10greg) [22:40:03] 10Deployment-Systems, 10Release-Engineering-Team (Kanban), 10Patch-For-Review: It shouldn't be possible to create WMF branches on master - https://phabricator.wikimedia.org/T175324#3637560 (10greg) a:03demon [22:42:22] 10Deployment-Systems, 10Project-Admins, 10User-greg: Further cleanup of #Deployment-Systems - https://phabricator.wikimedia.org/T126631#3637563 (10greg) [22:43:41] RECOVERY - Puppet errors on saucelabs-02 is OK: OK: Less than 1.00% above the threshold [0.0] [22:46:29] 10Deployment-Systems, 10Project-Admins, 10User-greg: Further cleanup of #Deployment-Systems - https://phabricator.wikimedia.org/T126631#3637567 (10greg) [22:57:50] 10Release-Engineering-Team (Kanban), 10MediaWiki-Release-Tools, 10Patch-For-Review: It shouldn't be possible to create WMF branches on master - https://phabricator.wikimedia.org/T175324#3637606 (10greg) [23:02:02] greg-g: legoktm does it look like that ticket is a wikidiff2 thing? [23:02:07] * addshore is nearly bed [23:12:43] addshore: no idea I just guessed. [23:13:00] I can't see the stack trace that was posted on the ticket [23:13:06] (On my phone) [23:14:52] Ooh, I'll have a look tommorrow [23:14:54] Bed now [23:19:07] 10Deployment-Systems, 10Architecture, 10Wikimedia-Developer-Summit-2016-Organization, 10Availability: WikiDev 16 working area: Software engineering - https://phabricator.wikimedia.org/T119032#3637730 (10greg) 05Open>03Resolved > daniel closed subtask T124504: Transition WikiDev '16 working areas into w... [23:31:49] 10Browser-Tests-Infrastructure, 10Release-Engineering-Team (Backlog), 10Deployments: Run @integration tests on new deploy branch creation - https://phabricator.wikimedia.org/T111545#3637816 (10greg) >>! In T111545#3298099, @zeljkofilipin wrote: > @greg replaced by [[ https://phabricator.wikimedia.org/tag/rel... [23:38:05] 10Release-Engineering-Team (Kanban), 10Deployments, 10Project-Admins, 10User-greg: Further cleanup of #Deployment-Systems - https://phabricator.wikimedia.org/T126631#3637821 (10greg) a:03greg Just did a few things: * renamed Deployment-Systems to Deployments * Made a #mediawiki-release-tools sub-project,... [23:45:15] 10Release-Engineering-Team (Kanban), 10User-greg: End of Q1 grooming - https://phabricator.wikimedia.org/T176523#3637859 (10greg) https://phabricator.wikimedia.org/daemon/bulk/view/1232/ [23:58:36] 10Release-Engineering-Team (Kanban), 10User-greg: End of Q1 grooming - https://phabricator.wikimedia.org/T176523#3638264 (10greg) https://phabricator.wikimedia.org/maniphest/query/Z26lXqwqqpu_/#R