[00:48:55] (03CR) 10Legoktm: "YES PLEASE THIS HAS BOTHERED ME FOREVER" [integration/docroot] - 10https://gerrit.wikimedia.org/r/378743 (owner: 10Krinkle) [04:07:54] Yippee, build fixed! [04:07:55] Project selenium-MultimediaViewer » safari,beta,OS X 10.9,BrowserTests build #522: 09FIXED in 11 min: https://integration.wikimedia.org/ci/job/selenium-MultimediaViewer/BROWSER=safari,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=OS%20X%2010.9,label=BrowserTests/522/ [04:16:30] Yippee, build fixed! [04:16:31] Project selenium-MultimediaViewer » chrome,beta,OS X 10.9,BrowserTests build #522: 09FIXED in 20 min: https://integration.wikimedia.org/ci/job/selenium-MultimediaViewer/BROWSER=chrome,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=OS%20X%2010.9,label=BrowserTests/522/ [04:16:55] Yippee, build fixed! [04:16:56] Project selenium-MultimediaViewer » firefox,beta,Linux,BrowserTests build #522: 09FIXED in 20 min: https://integration.wikimedia.org/ci/job/selenium-MultimediaViewer/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=BrowserTests/522/ [05:06:48] PROBLEM - Mediawiki Error Rate on graphite-labs is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [10.0] [05:13:17] RECOVERY - Puppet errors on integration-slave-docker-1001 is OK: OK: Less than 1.00% above the threshold [0.0] [05:36:49] PROBLEM - Mediawiki Error Rate on graphite-labs is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [10.0] [05:39:18] PROBLEM - Puppet errors on integration-slave-docker-1001 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [06:14:18] RECOVERY - Puppet errors on integration-slave-docker-1001 is OK: OK: Less than 1.00% above the threshold [0.0] [06:35:17] PROBLEM - Puppet errors on integration-slave-docker-1001 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [07:15:19] RECOVERY - Puppet errors on integration-slave-docker-1001 is OK: OK: Less than 1.00% above the threshold [0.0] [07:36:18] PROBLEM - Puppet errors on integration-slave-docker-1001 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [07:49:36] (03CR) 10Hashar: "> Upstreaming at https://review.openstack.org/#/c/505366/" [integration/docroot] - 10https://gerrit.wikimedia.org/r/373667 (https://phabricator.wikimedia.org/T174058) (owner: 10Thcipriani) [07:50:50] (03CR) 10Hashar: [C: 032] "Bubble bubble. Thank you Timo." [integration/docroot] - 10https://gerrit.wikimedia.org/r/378743 (owner: 10Krinkle) [07:51:32] (03Merged) 10jenkins-bot: zuul: Don't toggle panel when clicking Gerrit patch link [integration/docroot] - 10https://gerrit.wikimedia.org/r/378743 (owner: 10Krinkle) [07:51:38] (03CR) 10jenkins-bot: zuul: Don't toggle panel when clicking Gerrit patch link [integration/docroot] - 10https://gerrit.wikimedia.org/r/378743 (owner: 10Krinkle) [07:53:05] (03CR) 10Hashar: [C: 04-1] docker: composer image (032 comments) [integration/config] - 10https://gerrit.wikimedia.org/r/378531 (owner: 10Addshore) [07:53:13] (03PS3) 10Hashar: docker: composer image [integration/config] - 10https://gerrit.wikimedia.org/r/378531 (owner: 10Addshore) [07:55:24] (03CR) 10Hashar: [C: 031] docker: zuul-cloner image (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/375834 (owner: 10Addshore) [07:55:29] (03PS8) 10Hashar: docker: zuul-cloner image [integration/config] - 10https://gerrit.wikimedia.org/r/375834 (owner: 10Addshore) [07:56:23] (03CR) 10Hashar: [C: 032] docker: zuul-cloner image [integration/config] - 10https://gerrit.wikimedia.org/r/375834 (owner: 10Addshore) [07:58:16] (03Merged) 10jenkins-bot: docker: zuul-cloner image [integration/config] - 10https://gerrit.wikimedia.org/r/375834 (owner: 10Addshore) [07:58:28] (03PS4) 10Hashar: docker: composer image [integration/config] - 10https://gerrit.wikimedia.org/r/378531 (owner: 10Addshore) [08:16:27] 10Beta-Cluster-Infrastructure: Access to deployment-prep for sau226 - https://phabricator.wikimedia.org/T176213#3620429 (10Sau226) I understand that the beta cluster is for developers to test their code? Do correct me if I'm wrong but I believe that if a user repeatedly creates a page with a nonsense title and n... [08:17:26] (03CR) 10Hashar: [C: 04-1] "Looks fine but should be on top of a patch that provides the image wmfreleng/php:latest" [integration/config] - 10https://gerrit.wikimedia.org/r/378531 (owner: 10Addshore) [08:19:40] 10Beta-Cluster-Infrastructure: User rights request for clearing spam - https://phabricator.wikimedia.org/T176299#3620432 (10Anooprao) [08:23:34] (03PS3) 10Hashar: docker: base image for CI images [integration/config] - 10https://gerrit.wikimedia.org/r/378033 [08:24:09] (03CR) 10Hashar: "I have added the 'git' package to the proposed ci image https://gerrit.wikimedia.org/r/#/c/378033/2..3/dockerfiles/ci-jessie/Dockerfile" [integration/config] - 10https://gerrit.wikimedia.org/r/378534 (owner: 10Addshore) [08:32:00] PROBLEM - Puppet errors on deployment-aqs01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [08:40:51] PROBLEM - Puppet errors on deployment-aqs03 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [08:41:03] PROBLEM - Puppet errors on deployment-aqs02 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [08:46:02] (03CR) 10Hashar: [C: 04-1] "As addshore on contint1001:" [integration/config] - 10https://gerrit.wikimedia.org/r/378665 (https://phabricator.wikimedia.org/T167845) (owner: 10Hashar) [08:46:58] (03CR) 10Hashar: [C: 04-1] "Bah our fabfile.py has:" [integration/config] - 10https://gerrit.wikimedia.org/r/378665 (https://phabricator.wikimedia.org/T167845) (owner: 10Hashar) [08:53:24] (03PS3) 10Hashar: fab: reload zuul via systemd [integration/config] - 10https://gerrit.wikimedia.org/r/378665 (https://phabricator.wikimedia.org/T167845) [08:53:26] (03PS1) 10Hashar: fab: change sudo prefix [integration/config] - 10https://gerrit.wikimedia.org/r/379186 (https://phabricator.wikimedia.org/T167845) [08:54:45] (03CR) 10Hashar: "Can a non root try:" [integration/config] - 10https://gerrit.wikimedia.org/r/378665 (https://phabricator.wikimedia.org/T167845) (owner: 10Hashar) [08:55:00] addshore: fab deploy_zuul should be fixed with the above ^ [08:56:50] RECOVERY - Mediawiki Error Rate on graphite-labs is OK: OK: Less than 1.00% above the threshold [1.0] [09:00:46] PROBLEM - Puppet errors on deployment-puppetdb01 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [09:16:17] RECOVERY - Puppet errors on integration-slave-docker-1001 is OK: OK: Less than 1.00% above the threshold [0.0] [09:38:34] (03CR) 10Hashar: [C: 031] "I think setting HOME is sufficient :)" [integration/config] - 10https://gerrit.wikimedia.org/r/379186 (https://phabricator.wikimedia.org/T167845) (owner: 10Hashar) [09:40:09] (03CR) 10Hashar: [C: 032] docker: mediawiki-phan [integration/config] - 10https://gerrit.wikimedia.org/r/378533 (owner: 10Addshore) [09:40:46] RECOVERY - Puppet errors on deployment-puppetdb01 is OK: OK: Less than 1.00% above the threshold [0.0] [10:07:18] PROBLEM - Puppet errors on integration-slave-docker-1001 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [10:08:50] 10Beta-Cluster-Infrastructure: Access to deployment-prep for sau226 - https://phabricator.wikimedia.org/T176213#3620689 (10Sau226) https://deployment.wikimedia.beta.wmflabs.org/wiki/Seven_Awesome_Tips_About_Capital_One_Card_Log_In_From_Unlikely_Sources , https://deployment.wikimedia.beta.wmflabs.org/wiki/What_Ww... [10:16:05] 10Beta-Cluster-Infrastructure: User rights request for clearing spam - https://phabricator.wikimedia.org/T176299#3620432 (10Sau226) I get the feeling that no one will grant you rights until my issue ([[https://phabricator.wikimedia.org/T176213#3620429|phabricator link]]) is resolved as I granted you admin. Unfor... [10:44:46] 10Beta-Cluster-Infrastructure: Steward rights on the beta cluster - https://phabricator.wikimedia.org/T175555#3620737 (10Sau226) This change has been negated by actions as docummented in T176213. I have posted comments in my defense and am currently awaiting a response. [10:53:32] Does anyone can enable travis in https://travis-ci.org/wikimedia/mediawiki-extensions-DataTypes ? [11:32:11] 10Beta-Cluster-Infrastructure: User rights request for clearing spam - https://phabricator.wikimedia.org/T176299#3620857 (10Aklapper) How is this request different from T176140 ? [11:37:21] 10Beta-Cluster-Infrastructure: User rights request for clearing spam - https://phabricator.wikimedia.org/T176299#3620862 (10Anooprao) My admin rights were revoked locally since earlier rights were not processed through phabricator & @Aklapper this request differs this request is not for global use. [11:47:21] RECOVERY - Puppet errors on integration-slave-docker-1001 is OK: OK: Less than 1.00% above the threshold [0.0] [11:53:39] !log Reloading Zuul (Testing) [11:53:42] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [11:54:28] (03CR) 10Addshore: [C: 032] fab: reload zuul via systemd [integration/config] - 10https://gerrit.wikimedia.org/r/378665 (https://phabricator.wikimedia.org/T167845) (owner: 10Hashar) [12:02:22] addshore: !!!!! :) [12:02:28] sudo is painful [12:08:18] PROBLEM - Puppet errors on integration-slave-docker-1001 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [12:09:00] hashar: hahahaa [12:09:22] hashar: thanks for all the review of the container images! [12:09:34] * addshore is going to go and sort the rest out now [12:09:48] ;D [12:09:59] addshore: I have screwed up the dependency chain at some point [12:10:04] but iirc the php image can be pushed [12:10:13] I'll rebase that one now! [12:11:44] (03PS3) 10Addshore: docker: php7 base image [integration/config] - 10https://gerrit.wikimedia.org/r/378530 [12:11:49] hehhee hashar "That is clever :]" for the single update for apt [12:11:52] yeh :D saves such time! :D [12:37:19] hashar: ^^ the php one is ready then I guesss :D [12:38:04] I would be all for moving these to a different repo too, so that we can try and setup some CI jobs for it [12:38:25] could do a subtree split thing from /config too so we keep the history [12:38:46] addshore: thx for the work you are doing on all that btw.. [12:38:47] (03PS5) 10Addshore: docker: composer image [integration/config] - 10https://gerrit.wikimedia.org/r/378531 [12:39:58] (03PS3) 10Addshore: docker: mediawiki specific php image [integration/config] - 10https://gerrit.wikimedia.org/r/378532 [12:40:44] addshore: I am fine having them in integration/config :D [12:41:15] okay! :) I might put some thought into that [12:41:24] there must at least be some easy docker file linting or something [12:42:28] (03CR) 10Addshore: [C: 032] fab: change sudo prefix [integration/config] - 10https://gerrit.wikimedia.org/r/379186 (https://phabricator.wikimedia.org/T167845) (owner: 10Hashar) [12:42:52] addshore: https://gerrit.wikimedia.org/r/#/c/378530/3 is missing sury.gpg file :] And a DEBIAN_FRONTEND=noninteractive [12:45:19] (03CR) 10Hashar: [C: 04-1] "DEBIAN_FRONTEND=noninteractive" [integration/config] - 10https://gerrit.wikimedia.org/r/378530 (owner: 10Addshore) [12:45:40] addshore: can you amend it with the gpg key + tweak apt-get install ? :) [12:46:49] (03Merged) 10jenkins-bot: fab: change sudo prefix [integration/config] - 10https://gerrit.wikimedia.org/r/379186 (https://phabricator.wikimedia.org/T167845) (owner: 10Hashar) [12:47:07] (03CR) 10Addshore: "So, that depends how we want to go about it really." [integration/config] - 10https://gerrit.wikimedia.org/r/378534 (owner: 10Addshore) [12:47:27] hashar: yup. Also see my coment about the git one ^^ I'm really still not sure what the 'right' approach is [12:48:42] (03PS4) 10Addshore: docker: php7 base image [integration/config] - 10https://gerrit.wikimedia.org/r/378530 [12:48:44] hashar: ^^ done [12:49:03] the git container, I dont see the point :D [12:49:14] sounsd easier to just ship it in the base image [12:49:43] Yes, but, the 'git' commands in the jenkins jobs don't run in a container [12:49:59] they just run on the docker host. [12:50:31] ahoh yeah : [12:50:32] D [12:50:38] well we can just install git on the docker host? [12:50:50] yeh, thats the other way to go, and just run it there [12:50:59] I guess it is simpler that way [12:51:00] which I think is okay [12:51:12] although, it is also super easy to have the image for it :P [12:51:36] and then in theroy you can have different jobs running different version of git, not that you would porbably ever need that [12:51:49] but in principle, that would be the way you allow fully customizable jobs I guess [12:52:48] well as a first step, lets try to migrate a few jobs :] [12:52:50] also, for the phan job, I may be able to make it all as simple as running a small bash script before the bulk of the job, that sets up a bunch of bash aliases to commands to make them run in docker containers with the correct working dir mounted. so converting a job could literally be a one liner! [12:52:59] yup! all of these images are geared toward the phan job :) [12:53:09] which is good [12:53:56] (03CR) 10Hashar: [C: 032] docker: php7 base image [integration/config] - 10https://gerrit.wikimedia.org/r/378530 (owner: 10Addshore) [12:54:00] one more :D [12:54:11] addshore: can you docker push that wmfreleng/php image please ? :) [12:55:18] I'll push them all in a little bit :) [12:55:30] going to dash out for 5/10 mins now but ill be back! [12:56:15] (03Merged) 10jenkins-bot: docker: php7 base image [integration/config] - 10https://gerrit.wikimedia.org/r/378530 (owner: 10Addshore) [12:56:37] hashar: okay, actually still here! [12:57:05] the composer image is probably also ready to go out the door :) [13:00:13] yup [13:00:37] as to how we will port the jenkins jobs I have no clue [13:00:38] if we get this chain merged today I can update the phan docker job :D [13:00:40] wheeeeee [13:00:44] #proofofconcept [13:00:49] but I felt about having a container having everything [13:01:04] so you would docker run wmfreleng/mediawiki dothetests [13:01:14] or something like that, and the container would have all the things we need [13:01:19] but I dont know really [13:02:11] So I did some reading on docker CI stuff (I cant remember the links) but the right approach with containers in general seemed to be the classic, keep them responsible for 1 thing, or running 1 thing, which is why I'm intrigued to see how the phan one works [13:02:38] (03CR) 10Hashar: [C: 032] docker: composer image [integration/config] - 10https://gerrit.wikimedia.org/r/378531 (owner: 10Addshore) [13:02:42] mediawiki tests are a bit more complex than simply running phan though, because of things like "composer test" that actually does a bunch of stuff [13:02:48] not throught too deeply about that yet [13:03:29] but this is kind of sweet, for one of our WMDE things I'm trying to get a container setup for R linting :D https://gerrit.wikimedia.org/r/#/c/378831/ [13:04:14] yeah there is lame job that runs R tests [13:04:25] Yippee, build fixed! [13:04:25] Project selenium-Math » chrome,beta,Linux,BrowserTests build #520: 09FIXED in 23 sec: https://integration.wikimedia.org/ci/job/selenium-Math/BROWSER=chrome,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=BrowserTests/520/ [13:04:26] one of the trouble though is that R modules/dependencies often require a gcc compile [13:04:29] which is slow as hell [13:04:34] Yippee, build fixed! [13:04:35] Project selenium-Math » firefox,beta,Linux,BrowserTests build #520: 09FIXED in 33 sec: https://integration.wikimedia.org/ci/job/selenium-Math/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=BrowserTests/520/ [13:05:10] (03PS6) 10Hashar: docker: composer image [integration/config] - 10https://gerrit.wikimedia.org/r/378531 (owner: 10Addshore) [13:05:16] (03PS4) 10Addshore: docker: mediawiki specific php image [integration/config] - 10https://gerrit.wikimedia.org/r/378532 [13:05:18] hashar: yeh [13:05:23] (03CR) 10Hashar: [C: 032] docker: composer image [integration/config] - 10https://gerrit.wikimedia.org/r/378531 (owner: 10Addshore) [13:05:25] (03CR) 10jerkins-bot: [V: 04-1] docker: mediawiki specific php image [integration/config] - 10https://gerrit.wikimedia.org/r/378532 (owner: 10Addshore) [13:05:29] bah [13:05:30] also trying to figure out what packages the thing need is also a pain in the arse [13:05:55] hehe, I think we broke the chain again :D [13:06:00] ;D [13:06:24] (03PS5) 10Hashar: docker: mediawiki specific php image [integration/config] - 10https://gerrit.wikimedia.org/r/378532 (owner: 10Addshore) [13:06:48] right, back in 5 again now :D [13:07:07] (03Merged) 10jenkins-bot: docker: composer image [integration/config] - 10https://gerrit.wikimedia.org/r/378531 (owner: 10Addshore) [13:07:55] !log deployment-prep Ran cleanupSpam.php on deploymentwiki. Further testing with regards to ongoing development and updating of the script. [13:07:58] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [13:08:24] (03CR) 10Hashar: docker: mediawiki specific php image (033 comments) [integration/config] - 10https://gerrit.wikimedia.org/r/378532 (owner: 10Addshore) [13:13:16] RECOVERY - Puppet errors on integration-slave-docker-1001 is OK: OK: Less than 1.00% above the threshold [0.0] [13:16:11] 10Beta-Cluster-Infrastructure, 10Continuous-Integration-Infrastructure, 10Operations, 10Operations-Software-Development, and 2 others: Replace salt on integration and deployment-prep projects - https://phabricator.wikimedia.org/T176314#3621041 (10hashar) [13:22:08] I am off be back later tonight [13:22:15] addshore: feel free to publish the images :) [13:22:21] nothing we cant fix later on anyway [13:22:26] Yippee, build fixed! [13:22:26] Project selenium-MobileFrontend » chrome,beta,Linux,BrowserTests build #563: 09FIXED in 3 min 9 sec: https://integration.wikimedia.org/ci/job/selenium-MobileFrontend/BROWSER=chrome,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=BrowserTests/563/ [13:24:48] 10Beta-Cluster-Infrastructure: User rights request for clearing spam - https://phabricator.wikimedia.org/T176299#3621072 (10Sau226) Just close one of your requests as it makes it confusing for stewards @Anooprao . If I were you I'd close my local right one and use the global one as Aklapper suggested. [13:26:04] Yippee, build fixed! [13:26:04] Project selenium-MobileFrontend » firefox,beta,Linux,BrowserTests build #563: 09FIXED in 6 min 47 sec: https://integration.wikimedia.org/ci/job/selenium-MobileFrontend/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=BrowserTests/563/ [13:26:16] hasharAway: ack! :) [13:30:44] (03PS6) 10Addshore: docker: mediawiki specific php image [integration/config] - 10https://gerrit.wikimedia.org/r/378532 [13:31:08] (03CR) 10Addshore: [V: 031 C: 032] docker: mediawiki specific php image [integration/config] - 10https://gerrit.wikimedia.org/r/378532 (owner: 10Addshore) [13:31:19] 10Beta-Cluster-Infrastructure: User rights request for clearing spam - https://phabricator.wikimedia.org/T176299#3621088 (10Anooprao) [13:31:19] (03CR) 10Addshore: [C: 04-1] docker: git [integration/config] - 10https://gerrit.wikimedia.org/r/378534 (owner: 10Addshore) [13:31:21] 10Beta-Cluster-Infrastructure: Requesting Global Rights - https://phabricator.wikimedia.org/T176140#3621090 (10Anooprao) [13:32:27] 10Beta-Cluster-Infrastructure: Requesting Global Rights - https://phabricator.wikimedia.org/T176140#3614785 (10Anooprao) [13:32:29] 10Beta-Cluster-Infrastructure: User rights request for clearing spam - https://phabricator.wikimedia.org/T176299#3620432 (10Anooprao) [13:33:10] (03Merged) 10jenkins-bot: docker: mediawiki specific php image [integration/config] - 10https://gerrit.wikimedia.org/r/378532 (owner: 10Addshore) [13:33:31] 10Beta-Cluster-Infrastructure: User rights request for clearing spam - https://phabricator.wikimedia.org/T176299#3621111 (10Anooprao) 05duplicate>03Open [13:34:22] (03PS3) 10Addshore: docker: mediawiki-phan [integration/config] - 10https://gerrit.wikimedia.org/r/378533 [13:34:29] (03CR) 10Addshore: [C: 032] docker: mediawiki-phan [integration/config] - 10https://gerrit.wikimedia.org/r/378533 (owner: 10Addshore) [13:35:27] (03Merged) 10jenkins-bot: docker: mediawiki-phan [integration/config] - 10https://gerrit.wikimedia.org/r/378533 (owner: 10Addshore) [13:43:54] 10Continuous-Integration-Infrastructure (shipyard): build script should build dependant containers in order - https://phabricator.wikimedia.org/T176318#3621135 (10Addshore) [13:51:42] (03PS1) 10Addshore: docker: add ca-certificates to mediawiki-phan [integration/config] - 10https://gerrit.wikimedia.org/r/379230 [13:51:50] (03CR) 10Addshore: [V: 031 C: 032] docker: add ca-certificates to mediawiki-phan [integration/config] - 10https://gerrit.wikimedia.org/r/379230 (owner: 10Addshore) [13:52:56] (03Merged) 10jenkins-bot: docker: add ca-certificates to mediawiki-phan [integration/config] - 10https://gerrit.wikimedia.org/r/379230 (owner: 10Addshore) [13:59:11] https://www.irccloud.com/pastebin/ANaeGwdJ/ [13:59:13] meh [13:59:15] !log docker push docker.io/wmfreleng/php:v2017.09.20.13.40 & latest # built from master [13:59:15] !log docker push docker.io/wmfreleng/php-mediawiki:v2017.09.20.13.43 & latest # built from master [13:59:15] !log docker push docker.io/wmfreleng/zuul-cloner:v2017.09.20.13.44 & latest # built from master [13:59:15] !log docker push docker.io/wmfreleng/composer:v2017.09.20.13.44 & latest # built from master [13:59:15] !log docker push docker.io/wmfreleng/mediawiki-phan:v2017.09.20.13.49 & latest # built from master [13:59:19] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [13:59:22] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [13:59:25] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [13:59:28] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [13:59:35] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [14:00:04] addshore: spammmmmmmmm [14:02:18] 10Beta-Cluster-Infrastructure: Access to deployment-prep for sau226 - https://phabricator.wikimedia.org/T176213#3621190 (10Sau226) Thanks. Would it be possible for me to have even a subset of my old rights back or is it not possible? [14:09:17] PROBLEM - Puppet errors on integration-slave-docker-1001 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [14:39:07] Reedy: your a spam [14:44:17] RECOVERY - Puppet errors on integration-slave-docker-1001 is OK: OK: Less than 1.00% above the threshold [0.0] [14:56:07] 10Beta-Cluster-Infrastructure, 10Continuous-Integration-Infrastructure, 10Operations, 10Operations-Software-Development, and 2 others: Replace salt on integration and deployment-prep projects - https://phabricator.wikimedia.org/T176314#3621406 (10faidon) beta, CI and other WMCS VPS projects are not environ... [15:05:16] PROBLEM - Puppet errors on integration-slave-docker-1001 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [15:13:30] 10Beta-Cluster-Infrastructure, 10Continuous-Integration-Infrastructure, 10Operations, 10Operations-Software-Development, and 2 others: Replace salt on integration and deployment-prep projects - https://phabricator.wikimedia.org/T176314#3621041 (10demon) I don't really feel like nitpicking over projects or... [15:31:32] (03PS1) 10Reedy: Add tests for mediawiki/libs/PasswordBlacklist [integration/config] - 10https://gerrit.wikimedia.org/r/379250 [15:31:42] ^ Anyone about to merge and deploy that please? [15:38:40] Reedy: me? :D [15:38:47] feel free [15:38:52] I could never get the deploying setup :P [15:39:00] (03CR) 10Addshore: [C: 032] Add tests for mediawiki/libs/PasswordBlacklist [integration/config] - 10https://gerrit.wikimedia.org/r/379250 (owner: 10Reedy) [15:39:39] * addshore does it in docker now ;) [15:41:37] (03Merged) 10jenkins-bot: Add tests for mediawiki/libs/PasswordBlacklist [integration/config] - 10https://gerrit.wikimedia.org/r/379250 (owner: 10Reedy) [15:43:50] (03PS4) 10Addshore: fab: reload zuul via systemd [integration/config] - 10https://gerrit.wikimedia.org/r/378665 (https://phabricator.wikimedia.org/T167845) (owner: 10Hashar) [15:43:57] (03CR) 10Addshore: [C: 032] fab: reload zuul via systemd [integration/config] - 10https://gerrit.wikimedia.org/r/378665 (https://phabricator.wikimedia.org/T167845) (owner: 10Hashar) [15:45:35] (03Merged) 10jenkins-bot: fab: reload zuul via systemd [integration/config] - 10https://gerrit.wikimedia.org/r/378665 (https://phabricator.wikimedia.org/T167845) (owner: 10Hashar) [15:46:25] !log reloading zuul for https://gerrit.wikimedia.org/r/#/c/379250/ [15:46:29] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [15:46:32] Reedy: done [15:46:35] 10Scap (Tech Debt Sprint 2017-Q2), 10Patch-For-Review: Use git as transport mechanism for MediaWiki scap deploys - https://phabricator.wikimedia.org/T147938#3621556 (10mmodell) [15:46:37] <3 [15:50:17] mobileapps clogging stuff up again :( [15:56:01] yarp [15:56:01] (03PS2) 10Addshore: WIP DNM docker: alternate docker based phan job.. [integration/config] - 10https://gerrit.wikimedia.org/r/378535 [15:56:28] mhhhhm, well, the aliases didnt work, hmmm [15:57:47] * addshore thinks [16:04:22] * addshore will wait for hasharAway :) [16:24:52] PROBLEM - Puppet errors on deployment-mira is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [16:42:15] 10Beta-Cluster-Infrastructure: Access to deployment-prep for sau226 - https://phabricator.wikimedia.org/T176213#3621729 (10Legoktm) Based on https://en.wikipedia.beta.wmflabs.org/w/index.php?title=Special:Log/block&page=User%3ASelenium+user I think it's clear you don't understand one of the main purposes of the... [16:43:06] Reedy: https://integration.wikimedia.org/ci/view/Experimental/job/addshore-alias-test/9/console any ideas? :P [16:43:33] don't run x? :P [16:46:17] Reedy: well, i want aliases to exist for all of the docker run, so i can alias commands like composer to run in docker containers [16:46:31] maybe this isnt actually the right way to do it and I should insead rewrite the job templates [16:46:32] meh [16:51:43] maybe have something in the workspace in $PATH and then add little scripts there instead.... hmm. meh [17:03:16] 10Beta-Cluster-Infrastructure, 10Continuous-Integration-Infrastructure, 10Operations, 10Operations-Software-Development, and 2 others: Replace salt on integration and deployment-prep projects - https://phabricator.wikimedia.org/T176314#3621911 (10bd808) https://github.com/bd808/wikimedia-cloud-vps-hostgrou... [17:04:56] RECOVERY - Puppet errors on deployment-mira is OK: OK: Less than 1.00% above the threshold [0.0] [17:08:27] (03CR) 10Niedzielski: [C: 04-2] "It's fine if this a bad approach but please let me know how we should handle package-lock files over the next quarter or so." [integration/config] - 10https://gerrit.wikimedia.org/r/375405 (owner: 10Niedzielski) [17:53:25] 10Release-Engineering-Team (Kanban), 10Phabricator: Add support for task types - https://phabricator.wikimedia.org/T93499#3622155 (10MBinder_WMF) Will there be support for batch-updating existing tasks to a new type, and also for filtering boards on type? :) [18:09:17] 10Beta-Cluster-Infrastructure, 10Analytics-Kanban, 10Wikimedia-Stream, 10Patch-For-Review: Decom RCStream in Beta Cluster - https://phabricator.wikimedia.org/T172356#3495960 (10Krenair) >>! In T172356#3496434, @hashar wrote: > and a DNS entry `instance-deployment-stream.deployment-prep.wmflabs.org.` see h... [18:15:17] RECOVERY - Puppet errors on integration-slave-docker-1001 is OK: OK: Less than 1.00% above the threshold [0.0] [18:15:36] 10Release-Engineering-Team (Kanban), 10Phabricator: Add support for task types - https://phabricator.wikimedia.org/T93499#3622225 (10mmodell) @MBinder_WMF I don't think it's possible to change the type of a task, at least not easily. It could probably be done in bulk with a massive database migration but it mi... [18:20:20] 10Beta-Cluster-Infrastructure, 10Continuous-Integration-Infrastructure, 10Operations, 10Operations-Software-Development, and 2 others: Replace salt on integration and deployment-prep projects - https://phabricator.wikimedia.org/T176314#3622234 (10greg) For the record, the choice of tags is automatic by the... [18:20:22] 10Continuous-Integration-Config: Tests always fail in travis - https://phabricator.wikimedia.org/T176330#3622235 (10Framawiki) [18:30:38] i would like to create a paste in phab and set a custom access policy on it [18:31:37] i can select the "private paste (wmf-nda)", but i cant change the groups. can i not select another group .. custom is greyed out on purpose? [18:39:24] it seems someone disabled that [18:39:32] i wonder could the ability be brought back? [18:39:33] https://phabricator.wikimedia.org/transactions/editengine/paste.paste/ [18:39:38] twentyafterfour ^^ [18:40:42] https://phabricator.wikimedia.org/transactions/editengine/paste.paste/view/26/ [18:41:45] yea, ACK, it's a feature i used to have and use [18:42:30] and i am not allowed to create new forms [19:06:19] PROBLEM - Puppet errors on integration-slave-docker-1001 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [19:08:59] no_justification, i will reinstall gerrit as 2.13 on gerrit-test3 to see if slaves still works. [19:09:08] I was presously using gerrit 2.14. [19:09:39] Yeah, install the same as we do in production [19:10:07] ok /me does :) [19:20:34] (03PS1) 10Hashar: dockerfiles: allow prebuild.sh without executable bit [integration/config] - 10https://gerrit.wikimedia.org/r/379297 [19:21:14] (03CR) 10Hashar: dockerfiles: allow prebuild.sh without executable bit (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/379297 (owner: 10Hashar) [19:33:43] (03PS1) 10Hashar: dockerfiles: pass shellcheck [integration/config] - 10https://gerrit.wikimedia.org/r/379300 [19:46:17] RECOVERY - Puppet errors on integration-slave-docker-1001 is OK: OK: Less than 1.00% above the threshold [0.0] [19:47:01] (03PS1) 10Hashar: dockerfile: for phan do not run composer as root [integration/config] - 10https://gerrit.wikimedia.org/r/379303 [19:52:25] (03PS1) 10Hashar: dockerfiles: use a subshell instead of pushd/popd [integration/config] - 10https://gerrit.wikimedia.org/r/379304 [19:58:34] hi hashar [19:58:43] addshore: :] [19:58:46] sooo [19:58:51] https://integration.wikimedia.org/ci/view/Experimental/job/addshore-alias-test/9/console [19:59:04] I was trying to make the aliases work and havnt really figured out how to do it in jenkins [19:59:05] poor x [19:59:30] aliases like alias git="docker run git" ? [19:59:32] basically for a job I was to define a set of command aliases at the start and for them to be used throughout, but it should be a per job thing [19:59:35] hashar: yup [20:00:07] that avoids having to rewrite the things like the slave-scripts that call composer to have both a docker and a non docker version etc [20:01:23] try to shopt expand_aliaes before sourcing ? [20:01:56] also "alias" should give you a list of them [20:02:12] oh no wait hm [20:02:23] (03CR) 10Addshore: dockerfiles: allow prebuild.sh without executable bit (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/379297 (owner: 10Hashar) [20:02:37] (03CR) 10Addshore: [C: 031] "I'll verify it before merging though" [integration/config] - 10https://gerrit.wikimedia.org/r/379300 (owner: 10Hashar) [20:02:55] Yeh, I did some googling and did the expand_aliaes [20:03:01] but the aliases dont seem to stick [20:03:07] even if I add them to .bashrc or .bashprofile [20:03:16] I guess the .bash_profile is not loaded ? [20:03:27] .bashrc bails out early if it is a non interactive shell, so adding there wouldnt work [20:03:36] maybe something similar happens with bash profile :/ [20:04:27] the invocation of bash and the rc/profile files always confused me [20:04:56] the man page should have all the info,but iirc it is an hour long to understand what is going on [20:05:26] (03CR) 10Addshore: dockerfile: for phan do not run composer as root (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/379303 (owner: 10Hashar) [20:05:54] When bash is started non-interactively [... stuff that does not mention either profile or rc files ...] [20:06:02] so I guess when it is non interactive it loads nothing [20:06:18] BUT the manual mentions $BASH_ENV which can be set to a file to load [20:06:25] and then it does: if [ -n "$BASH_ENV" ]; then . "$BASH_ENV"; fi [20:06:39] crazy :D [20:06:51] (03CR) 10Addshore: [C: 032] dockerfiles: use a subshell instead of pushd/popd [integration/config] - 10https://gerrit.wikimedia.org/r/379304 (owner: 10Hashar) [20:07:14] oooh [20:07:18] PROBLEM - Puppet errors on integration-slave-docker-1001 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [20:07:18] okay, let me try that hashar [20:07:47] hmm, in jjb is there a way to set env vars for a job? [20:08:02] i assume yes :P [20:08:27] (03PS1) 10Chad: Updates for REL1_30 [tools/release] - 10https://gerrit.wikimedia.org/r/379307 [20:08:29] (03CR) 10Chad: [C: 032] Updates for REL1_30 [tools/release] - 10https://gerrit.wikimedia.org/r/379307 (owner: 10Chad) [20:09:54] addshore: in jenkins, that is usually done by adding a parameter to the job. It ends up being exposed as an env variable to all shells [20:10:04] okay [20:10:13] but that is becoming dirty :] [20:10:14] "This project is parameterised" tick box then i guess [20:10:17] (03Merged) 10jenkins-bot: dockerfiles: use a subshell instead of pushd/popd [integration/config] - 10https://gerrit.wikimedia.org/r/379304 (owner: 10Hashar) [20:10:26] another way is to add them in the PATH somewhere, eg /usr/local/bin [20:10:27] (03Merged) 10jenkins-bot: Updates for REL1_30 [tools/release] - 10https://gerrit.wikimedia.org/r/379307 (owner: 10Chad) [20:11:06] Yeh, I was thinking path [20:11:13] but it should be a per build / workspace thing really [20:11:43] so that, say, different jobs and run different versions of the phan image etc, so one for phan 0.8 and one for 0.8.5 etc [20:13:21] Reedy: does https://phabricator.wikimedia.org/P6028 mean there's any problem? [20:16:26] PATH=/usr/local/bin:/usr/bin:/bin:/usr/games [20:18:44] meh, cant make the parameterization work, cant see the env var in the build :/ [20:19:41] addshore: then ship different containers? :) [20:19:54] but that becomes messy [20:20:17] https://integration.wikimedia.org/ci/view/Experimental/job/addshore-alias-test/configure [20:20:20] so that should set the env var? [20:20:43] phan:0.8+v2017.09.20.18.20 [20:20:53] phan:0.8.5+v2017.09.20.18.19 [20:21:09] oh wait, i need to do build with parameters now! [20:21:56] oooh, I think I might have it [20:22:58] hashar: https://integration.wikimedia.org/ci/view/Experimental/job/addshore-alias-test/18/console [20:23:20] !!!! [20:23:55] tidier one https://integration.wikimedia.org/ci/view/Experimental/job/addshore-alias-test/19/console [20:24:00] what I thought of was to run the whole build/scripts in a single container [20:24:23] I am not so sure it is a good idea to use the docker host to run bunch of commands. We will handle with the same trouble we had with the permanent slaves [20:24:40] with bunch of files leaking, nasty race conditions between jobs etc [20:24:40] yeh, I mean, right now I think I can see both sides, which I was I want to try this for the pahn job [20:24:49] yeah definitely [20:25:02] it is worth experimenting. But I am pretty sure it is going to doom us down the line [20:25:40] the whole thing really is that docker is not really meant to run multiple process either serially or in parallel [20:26:00] well at least the tool chain does not seem to fit that use case [20:34:26] (03PS3) 10Addshore: WIP DNM docker: alternate docker based phan job.. [integration/config] - 10https://gerrit.wikimedia.org/r/378535 [20:34:31] (03Abandoned) 10MarcoAurelio: Whitelist Dvorapa on Zuul CI [integration/config] - 10https://gerrit.wikimedia.org/r/375765 (owner: 10MarcoAurelio) [20:40:53] (03CR) 10Hashar: dockerfile: for phan do not run composer as root (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/379303 (owner: 10Hashar) [20:41:16] hashar: https://integration.wikimedia.org/ci/job/mwext-php70-phan-jessie-docker/13/console [20:41:42] (03PS4) 10Addshore: WIP DNM docker: alternate docker based phan job.. [integration/config] - 10https://gerrit.wikimedia.org/r/378535 [20:41:43] looks like the proof on concept actually works [20:41:56] /srv/jenkins-workspace/workspace/mwext-php70-phan-jessie-docker/.bash_docker_aliases: line 5: unexpected EOF while looking for matching `'' [20:41:56] :D [20:42:11] yeh, stupid me missed a quote [20:42:15] ah yeah good [20:42:23] I see zuul-cloner is around [20:42:35] oh yeh, Unable to read clone map file at /srv/deployment/integration/slave-scripts/etc/zuul-clonemap.yaml. [20:42:41] stupid different location [20:43:14] (03PS2) 10Hashar: dockerfile: for phan do not run composer as root [integration/config] - 10https://gerrit.wikimedia.org/r/379303 [20:43:19] install -d /srv/phan -o phan -g phan [20:43:29] or a single cmd to (mkdir -p && chown) [20:43:30] ooooh [20:43:30] ! [20:43:41] Debian packages are using install a lot [20:43:49] and I made myself a promise to eventually use it one day [20:44:50] + alias docker-sudo='docker run -it --user root --entrypoint /bin/bash' [20:45:54] (03CR) 10Hashar: "It managed to build an image and /srv/phan stuff belong to phan (though ideally all your base would belong to us)" [integration/config] - 10https://gerrit.wikimedia.org/r/379303 (owner: 10Hashar) [20:46:17] addshore: I will tackle the tox container tomorrow [20:47:14] and hashar in terms of keeping working space clean, using my method it could simply make docker volumes and use them throughout throwing it away after [20:47:17] RECOVERY - Puppet errors on integration-slave-docker-1001 is OK: OK: Less than 1.00% above the threshold [0.0] [20:49:23] addshore: potentially yeah :] [21:08:18] PROBLEM - Puppet errors on integration-slave-docker-1001 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [21:15:33] hashar: and I think whichever way we end up deciding is right, the base images I have made will be usefull :) [21:18:43] there is no git cache for core for zuul on the docker hosts :< [21:18:43] :P [21:20:17] 10Beta-Cluster-Infrastructure, 10Continuous-Integration-Infrastructure, 10Operations-Software-Development, 10Technical-Debt: Replace salt on integration and deployment-prep projects - https://phabricator.wikimedia.org/T176314#3622910 (10greg) leaving ops-software-dev as this is about cumin (though that may... [21:20:22] (03Restored) 10MarcoAurelio: Whitelist Dvorapa on Zuul CI [integration/config] - 10https://gerrit.wikimedia.org/r/375765 (owner: 10MarcoAurelio) [21:20:26] (03PS7) 10MarcoAurelio: Whitelist Dvorapa on Zuul CI [integration/config] - 10https://gerrit.wikimedia.org/r/375765 [21:26:47] no_justification it works for me [21:26:48] :) [21:26:55] i wonder is the slave running 2.13.9? [21:27:01] since prod is running 2.13.8 [21:27:05] [2017-09-20 21:26:07,767] [main] INFO com.google.gerrit.sshd.SshDaemon : Started Gerrit SSHD-CORE-1.2.0 on *:29418 [21:27:05] [2017-09-20 21:26:07,768] [main] INFO com.google.gerrit.pgm.Daemon : Gerrit Code Review 2.13.9-2-g99a8c8bc51-dirty ready [21:27:16] mutante ^^ [21:28:41] RUN_ARGS = -Xloggc:/srv/gerrit/jvmlogs/jvm_gc.%p.log -XX:+PrintGCApplicationStoppedTime -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintTenuringDistribution -XX:+PrintGCCause -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=10 -XX:GCLogFileSize=2M -Xmx4g -jar /var/lib/gerrit2/review_site/bin/gerrit.war daemon -d /var/lib/gerrit2/review_site --slave [21:29:49] addshore: re lack of git cache, yeah the idea was to ship the git repos in the image or eventually mount a volume [21:30:00] addshore: I cant remember which got picked for the operationspuppet image [21:30:52] wmfreleng/operations-puppet latest b8132d66860e 2 hours ago 789MB [21:30:52] I guess it has a clone :] [21:33:21] meh, /etc/passwd doesnt even have the jenkins-deploy user in it [21:33:43] hashar: indeed, but the operations-puppet image doesnt use zuul-cloner at al;l [21:34:03] yup [21:34:25] I think it worked more or less by using a volume like -v /srv/git:/srv/git:ro [21:34:39] git clone would not hardlink but do a straight copy iof the files [21:34:47] but that was fast enough in the few tests I ran [21:35:19] eg core is just a pack file of 256MB + a few other files [21:35:40] so that is reasonsably fast [21:36:47] * addshore really doesnt like packing the code in the image :( it makes it so bloated [21:37:24] i guess the caching between test runs should be done outside of the container, probably similar to what travis CI does [21:37:31] 10Beta-Cluster-Infrastructure, 10Continuous-Integration-Infrastructure, 10Operations-Software-Development, 10Technical-Debt: Replace salt on integration and deployment-prep projects - https://phabricator.wikimedia.org/T176314#3622953 (10Volans) @greg no, that's the right one, cumin it's an additional hasht... [21:38:49] if you strip all of the puppet specific stuff out of the puppet docker file and fin file, you essentially have a "bundle-rake-test" image, the code being tested could then be mounted from somewhere. [21:38:57] still not convinced I like either one, heh [21:39:04] Might give it a rest for today [21:39:04] :]]] [21:39:13] we can try each approach [21:39:40] and maybe I can write something about the caches we ahve currently [21:39:49] hashar: that would be nice! :D [21:40:08] so, why isnt the jenkins-deploy user in /etc/passwd? [21:40:33] oh and by the way, if we mount /srv/git as a read only volume [21:40:43] thats the thing that just tripped me up while playing with my phan stuff [21:40:43] there is a way for git to just refer to those objects without copying any [21:40:50] (git clone --reference does that) [21:41:06] jenkins-deploy is in WMCS LDAP [21:41:48] we should get rid of the hardcoded uid/gid in the images, but I havent looked at why they are needed [21:41:49] ;D [21:42:19] basically i hit this issue https://stackoverflow.com/questions/42426325/use-git-in-jenkins-pipeline-with-docker-agent#42531164 [21:43:04] file permissions, so, the issue i just had is my zuul-cloner image runs as the zuul-cloner user, this doesnt exist outside the container / cant write to th workspace so cant actually do anything, haha [21:43:19] RECOVERY - Puppet errors on integration-slave-docker-1001 is OK: OK: Less than 1.00% above the threshold [0.0] [21:43:35] but if I pass in -u jenkins-deploy on the docker run command, or the uid and gid, git fails as the user doesnt exist in /etc/passwd [21:43:57] addshore: can you -u with the numerical uid ? [21:44:10] for the ops-puppet image this isnt an issue as the -u jenkins-deploy user is specifically created in the image, and thus exists in the /etc/passwd file for the image [21:44:38] -u, --user string Username or UID (format: [:]) [21:44:41] but, thus, git only works if you create the -u jenkins-deploy user in the image (or maybe fake /etc/passwd [21:44:43] yeah try passing the current use ruid? [21:44:52] hashar: yeh, but git still complains if the user isnt in /etc/passwd [21:45:03] and as it isnt on the host machine i also cant do -v \"//etc/passwd://etc/passwd\" to just fix it [21:45:29] docker run --user "$(id --user)":"$(id --group)" [21:45:40] or what not. I dont know really [21:45:46] wont work using git :/ [21:45:49] * addshore finds an example [21:46:06] but yeah I can see how it would fail :D [21:47:01] addshore: I gotta head to bed anyway . I have a long day tomorrow [21:47:27] night! [21:47:51] but in short, using docker as one off app to mess on the host: that seems like quite challenging :D [21:48:42] * addshore is gonna read https://wiki.jenkins.io/display/JENKINS/Docker+Plugin [21:49:47] have sweet reads! [22:20:43] (03PS1) 10Chad: More 1.30 fixes [tools/release] - 10https://gerrit.wikimedia.org/r/379423 [22:20:45] (03CR) 10Chad: [C: 032] More 1.30 fixes [tools/release] - 10https://gerrit.wikimedia.org/r/379423 (owner: 10Chad) [22:21:56] (03Merged) 10jenkins-bot: More 1.30 fixes [tools/release] - 10https://gerrit.wikimedia.org/r/379423 (owner: 10Chad) [22:24:22] (03PS1) 10Chad: Nobody uses the smw bundle from this script anymore [tools/release] - 10https://gerrit.wikimedia.org/r/379424 [22:24:24] (03CR) 10Chad: [C: 032] Nobody uses the smw bundle from this script anymore [tools/release] - 10https://gerrit.wikimedia.org/r/379424 (owner: 10Chad) [22:26:48] 10Scap, 10HHVM: [scap] Compile HHVM bytecode cache as deployment step - https://phabricator.wikimedia.org/T66272#3623059 (10MaxSem) 05Open>03declined Not happening due to {T176370}. [22:27:13] (03Merged) 10jenkins-bot: Nobody uses the smw bundle from this script anymore [tools/release] - 10https://gerrit.wikimedia.org/r/379424 (owner: 10Chad) [23:09:18] PROBLEM - Puppet errors on integration-slave-docker-1001 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [23:24:12] mutante: you should be able to make a private paste now [23:24:51] twentyafterfour: oh :) thanks [23:25:10] twentyafterfour: i am debugging on gerrit2001, and i think i just found out finally why [23:25:21] in a minute , heh [23:36:47] Notice: /Stage[main]/Phabricator::Aphlict/Base::Service_unit[aphlict]/Service[aphlict]/ensure: ensure changed 'stopped' to 'running' [23:36:50] Info: /Stage[main]/Phabricator::Aphlict/Base::Service_unit[aphlict]/Service[aphlict]: Unscheduling refresh on Service[aphlict] [23:38:41] that's the Phab notification service, for the first time ever [23:38:48] starting in prod