[00:09:17] <shinken-wm>	 PROBLEM - Puppet errors on integration-slave-docker-1001 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0]
[00:28:55] <wikibugs_>	 10Continuous-Integration-Config, 10VisualEditor, 10VisualEditor-ContentEditable: VisualEditor IME test failing in CI on pull-through, but not locally or in the source repo? - https://phabricator.wikimedia.org/T176453#3626092 (10Jdforrester-WMF)
[01:14:16] <shinken-wm>	 RECOVERY - Puppet errors on integration-slave-docker-1001 is OK: OK: Less than 1.00% above the threshold [0.0]
[01:30:53] <wikibugs_>	 (03CR) 10Thcipriani: "Looks like a good start: container builds, is a good size, and has all needed dependencies. There are a few considerations for the /run.sh" (033 comments) [integration/config] - 10https://gerrit.wikimedia.org/r/379479 (owner: 10Legoktm)
[01:56:18] <wikibugs_>	 (03CR) 10Legoktm: [WIP] Add dockerfile for 'mediawiki-core-phpcs' job (033 comments) [integration/config] - 10https://gerrit.wikimedia.org/r/379479 (owner: 10Legoktm)
[02:10:17] <shinken-wm>	 PROBLEM - Puppet errors on integration-slave-docker-1001 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0]
[03:45:16] <shinken-wm>	 RECOVERY - Puppet errors on integration-slave-docker-1001 is OK: OK: Less than 1.00% above the threshold [0.0]
[04:36:18] <shinken-wm>	 PROBLEM - Puppet errors on integration-slave-docker-1001 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0]
[05:26:07] <kart_>	 Looks like, https://integration.wikimedia.org/zuul/ is stuck?
[05:39:52] <shinken-wm>	 PROBLEM - Mediawiki Error Rate on graphite-labs is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [10.0]
[05:58:42] <wikibugs_>	 10Beta-Cluster-Infrastructure: Access to deployment-prep for sau226 - https://phabricator.wikimedia.org/T176213#3626226 (10Sau226) @greg The bot is set up so I need to program it and then manually trigger it in order for it to run. Check the enwiki test server to find out an example of the (lots of disused) page...
[05:59:50] <shinken-wm>	 PROBLEM - Mediawiki Error Rate on graphite-labs is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [10.0]
[06:42:50] <wikibugs_>	 10Beta-Cluster-Infrastructure: Access to deployment-prep for sau226 - https://phabricator.wikimedia.org/T176213#3626234 (10greg) >>! In T176213#3626226, @Sau226 wrote: > @greg The bot is set up so I need to program it and then manually trigger it in order for it to run.  Just to be clear: Do NOT enable or use th...
[06:44:13] <greg-g>	 kart_: looks like zuul, yeah :/
[06:44:15] <greg-g>	 has<tab>
[06:45:30] <greg-g>	 !log Zuul is stuck, no jobs are processing
[06:45:34] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[06:56:57] <greg-g>	 I pinged _joe_ for help, he'll get to it soon
[06:57:20] <greg-g>	 !log pinged an opsen, hopefully they'll restart zuul shortly
[06:57:24] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[06:58:10] <greg-g>	 I can't ssh in for some reason:
[06:58:11] <greg-g>	 greg@x1  ~ % ssh contint1001.eqiad.wmnet   [255] 9 2551 23:51:22 Thu 21.09.2017
[06:58:14] <greg-g>	 Enter passphrase for key '/home/greg/.ssh/id_ed25519.wmfprod': 
[06:58:17] <greg-g>	 channel 0: open failed: administratively prohibited: open failed
[06:58:20] <greg-g>	 stdio forwarding failed
[06:58:22] <greg-g>	 ssh_exchange_identification: Connection closed by remote host
[07:04:50] <greg-g>	 !log deleting stuck mediawiki-core-jsduck-publish jobs in Jenkins UI
[07:04:55] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[07:10:20] <kart_>	 greg-g: thanks!
[07:13:44] <greg-g>	 !log some jsduck jobs are running now, serially, for the backlogged queue. Unsure of starved jobs (integration-config-qa, pywikibot-beta-cluster, etc)
[07:13:48] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[07:46:17] <shinken-wm>	 RECOVERY - Puppet errors on integration-slave-docker-1001 is OK: OK: Less than 1.00% above the threshold [0.0]
[07:47:41] <wikibugs_>	 10Release-Engineering-Team, 10Operations, 10Phabricator, 10Patch-For-Review: The aphlict systemd unit needs to be rewritten from scratch - https://phabricator.wikimedia.org/T176392#3626266 (10Paladox) The patch has stalled and dosent look like it will move along, I guess we should change the priority to no...
[08:22:55] <mobrovac>	 ssh: Could not resolve hostname bastion.wmflabs.org: Name or service not known
[08:22:57] <mobrovac>	 known?
[08:31:59] <shinken-wm>	 PROBLEM - Puppet errors on deployment-aqs01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0]
[08:34:49] <shinken-wm>	 RECOVERY - Mediawiki Error Rate on graphite-labs is OK: OK: Less than 1.00% above the threshold [1.0]
[08:40:50] <shinken-wm>	 PROBLEM - Puppet errors on deployment-aqs03 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0]
[08:41:04] <shinken-wm>	 PROBLEM - Puppet errors on deployment-aqs02 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0]
[08:42:16] <shinken-wm>	 PROBLEM - Puppet errors on integration-slave-docker-1001 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0]
[08:44:32] <hashar>	 !log Upgraded docker on integration-slave-docker-1001 and integration-slave-docker-1002 - T176267
[08:44:37] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[08:44:38] <stashbot>	 T176267: Upgrade docker on integration-slave-docker-* - https://phabricator.wikimedia.org/T176267
[08:49:49] <kart_>	 hashar: how can we fix errors like: https://integration.wikimedia.org/ci/job/npm-node-6-jessie/17115/console - anything from ourside needed?
[08:50:01] <kart_>	 hashar: or need to wait till those npms are updated?
[08:52:16] <shinken-wm>	 RECOVERY - Puppet errors on integration-slave-docker-1001 is OK: OK: Less than 1.00% above the threshold [0.0]
[09:11:53] <hashar>	 kart_: hello
[09:12:12] <hashar>	 kart_: looks like the test fails because some dependencies have security issues
[09:12:24] <hashar>	 but the CI job itself looks to be running just fine
[09:13:27] <hashar>	 kart_: and it seems that is caused by tough-cookie  which is a dependency of jsdom@10.1.0
[09:13:32] <hashar>	 which most probably should be updated :D
[09:29:36] <wikibugs_>	 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban), 10Patch-For-Review: Upgrade docker on integration-slave-docker-* - https://phabricator.wikimedia.org/T176267#3619405 (10hashar) a:03hashar
[09:30:04] <wikibugs_>	 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban), 10Patch-For-Review: Upgrade docker on integration-slave-docker-* - https://phabricator.wikimedia.org/T176267#3619405 (10hashar) p:05Triage>03Normal
[09:49:53] <wikibugs_>	 (03PS1) 10Addshore: Add AdvancedSearch to make-wmf-branch [tools/release] - 10https://gerrit.wikimedia.org/r/379710
[10:04:07] <wikibugs_>	 (03CR) 10Addshore: [C: 04-1] "not yet" [tools/release] - 10https://gerrit.wikimedia.org/r/379710 (owner: 10Addshore)
[10:06:42] <hashar>	 !log deployement-salt02 migrated hiera config from wikitech to horizon. Removed the class role::deployment::salt_masters 
[10:06:46] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[10:20:48] <shinken-wm>	 RECOVERY - Puppet errors on deployment-salt02 is OK: OK: Less than 1.00% above the threshold [0.0]
[10:48:54] <wikibugs_>	 10Release-Engineering-Team, 10Operations, 10Phabricator, 10Patch-For-Review: The aphlict systemd unit needs to be rewritten from scratch - https://phabricator.wikimedia.org/T176392#3626648 (10Joe) Thanks to @Paladox work on this, the aphlict service unit now handles correctly the software.  I am going to m...
[10:49:16] <wikibugs_>	 10Release-Engineering-Team, 10Operations, 10Phabricator, 10Patch-For-Review: The aphlict systemd unit needs to be rewritten from scratch - https://phabricator.wikimedia.org/T176392#3626649 (10Joe) 05Open>03Resolved a:03Paladox
[11:18:11] <kart_>	 hashar: let me try. Since cxserver dependency listed as ^10.1.0 it should self-upgrade, but that's not happening.
[11:50:40] <wikibugs_>	 10Release-Engineering-Team, 10Operations, 10Phabricator, 10Patch-For-Review: The aphlict systemd unit needs to be rewritten from scratch - https://phabricator.wikimedia.org/T176392#3626762 (10Paladox) @Joe thanks :)  Yeh we can remove Ubuntu / upstart support.
[11:54:51] <kart_>	 hashar: looks they still need to update: https://github.com/tmpvar/jsdom/pull/1985
[12:04:09] <hashar>	 kart_: maybe you can add that new version as a dependency of cxserver ?
[12:04:21] <hashar>	 but I am afraid npm would bring the old version for jsdom :(
[12:04:31] <kart_>	 hashar: yes.
[12:04:40] <kart_>	 hashar: I'll wait for while.
[12:04:57] <hashar>	 or whatever fails the security test might have a way to whitelist a module?
[12:06:50] <hashar>	 kart_: that comes from node-inspector -> nsp
[12:09:17] <kart_>	 hashar: can be done via npmrc: https://github.com/salesforce/tough-cookie/issues/92#issuecomment-331065539
[12:09:26] <kart_>	 hashar: i see.
[12:09:32] <kart_>	 Let me check that too.
[12:10:22] <hashar>	 kart_: and specially  scripts: test: "mocha && nsp check"
[12:10:48] <hashar>	 kart_: which leads me to https://www.npmjs.com/package/nsp#exceptions  :D
[12:11:12] <hashar>	 it is probably terrible to ignore a security issue, but if you get a task filled it is probably fine
[12:12:56] <wikibugs_>	 (03PS2) 10Legoktm: [WIP] Add dockerfile for 'mediawiki-core-phpcs' job [integration/config] - 10https://gerrit.wikimedia.org/r/379479
[12:12:58] <wikibugs_>	 (03CR) 10Legoktm: [WIP] Add dockerfile for 'mediawiki-core-phpcs' job (032 comments) [integration/config] - 10https://gerrit.wikimedia.org/r/379479 (owner: 10Legoktm)
[12:19:34] <wikibugs_>	 10Gerrit: Migrate to NoteDb - https://phabricator.wikimedia.org/T174034#3626843 (10Paladox) We cannot make a notedb file in puppet we have to let gerrit create it once the migration is complete and move the configs to gerrit.config then delete notedb.config.
[12:22:54] <wmf-insecte>	 Yippee, build fixed!
[12:22:54] <wmf-insecte>	 Project selenium-GettingStarted » firefox,beta,Linux,BrowserTests build #533: 09FIXED in 53 sec: https://integration.wikimedia.org/ci/job/selenium-GettingStarted/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=BrowserTests/533/
[13:16:36] <kart_>	 hashar: will avoid adding security exception. Lets wait.
[13:22:18] <wikibugs_>	 (03PS5) 10Hashar: tox Dockefile [integration/config] - 10https://gerrit.wikimedia.org/r/377337 (owner: 10Addshore)
[13:26:10] <wikibugs_>	 (03Draft1) 10MarcoAurelio: Archive Extension:WikiTwidget [integration/config] - 10https://gerrit.wikimedia.org/r/379754 (https://phabricator.wikimedia.org/T148826)
[13:26:15] <wikibugs_>	 (03PS2) 10MarcoAurelio: Archive Extension:WikiTwidget [integration/config] - 10https://gerrit.wikimedia.org/r/379754 (https://phabricator.wikimedia.org/T148826)
[13:29:01] <addshore>	 hashar: gonna take another look at all the docker things now :)
[13:29:31] <addshore>	 We may have decided using the user "nobody" in the images is the right thing to do
[13:32:25] <wikibugs_>	 (03CR) 10Zoranzoki21: [C: 031] Archive Extension:WikiTwidget [integration/config] - 10https://gerrit.wikimedia.org/r/379754 (https://phabricator.wikimedia.org/T148826) (owner: 10MarcoAurelio)
[13:32:27] <hashar>	 addshore: I would follow the recommandation :]
[13:32:41] <hashar>	 I guess there are bunch of patches that have to be amended !
[13:37:26] <hashar>	 and potentially we could use eatmydata to speed up the packages installation :D
[13:37:44] <addshore>	 whats eatmydata? :D
[13:38:12] <hashar>	 it is a small lib that is prepended to LD_LOADPATH  which hihack fsync() and other system calls
[13:38:27] <hashar>	 so that instead of syncing writes to the disk does... nothing!
[13:38:41] <hashar>	 so you save up all the delay waiting for the FS to reply OK I REALLY WROTE THE DATA ON DISK
[13:39:42] <wikibugs_>	 (03PS6) 10Hashar: tox Dockefile [integration/config] - 10https://gerrit.wikimedia.org/r/377337 (owner: 10Addshore)
[13:39:44] <wikibugs_>	 (03PS2) 10Hashar: dockerfiles: support for http_proxy [integration/config] - 10https://gerrit.wikimedia.org/r/379507
[13:40:15] <wikibugs_>	 (03CR) 10Hashar: [C: 032] "Added DEBIAN_FRONTEND=noninteractive for apt-get install." [integration/config] - 10https://gerrit.wikimedia.org/r/377337 (owner: 10Addshore)
[13:41:19] <wikibugs_>	 (03Merged) 10jenkins-bot: tox Dockefile [integration/config] - 10https://gerrit.wikimedia.org/r/377337 (owner: 10Addshore)
[13:41:59] <wikibugs_>	 10Release-Engineering-Team (Backlog), 10Cleanup, 10Repository-Admins, 10Patch-For-Review, 10User-MarcoAurelio: Deprecate unmaintained/inactive WikiTwidget extension - https://phabricator.wikimedia.org/T148826#3627058 (10MarcoAurelio) I need someone within #release-engineering-team to do the GitHub stuff....
[13:46:19] <wmf-insecte>	 Project selenium-VisualEditor » firefox,beta,Linux,BrowserTests build #530: 04FAILURE in 2 min 18 sec: https://integration.wikimedia.org/ci/job/selenium-VisualEditor/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=BrowserTests/530/
[13:58:52] <wikibugs_>	 (03PS3) 10Hashar: dockerfiles: support for http_proxy [integration/config] - 10https://gerrit.wikimedia.org/r/379507
[13:59:19] <wikibugs_>	 (03PS4) 10Hashar: dockerfiles: support for a build.env file and http_proxy [integration/config] - 10https://gerrit.wikimedia.org/r/379507
[13:59:44] <wikibugs_>	 (03CR) 10Hashar: "Added support to have build.sh to load an optional build.env file." [integration/config] - 10https://gerrit.wikimedia.org/r/379507 (owner: 10Hashar)
[14:03:08] <wikibugs_>	 (03PS1) 10Addshore: WIP docker: use nobody for operations-puppet [integration/config] - 10https://gerrit.wikimedia.org/r/379762
[14:03:27] <addshore>	 hashar: ^^ /usr/lib/ruby/vendor_ruby/bundler/spec_set.rb:92:in `block in materialize': Could not find rake-12.0.0 in any of the sources (Bundler::GemNotFound) :(
[14:04:49] <addshore>	 also hashar  for https://gerrit.wikimedia.org/r/#/c/379507/4 im guessing there is a docker image for running an apt-cacher-ng we can just use? :D or build into the build script itself?
[14:09:19] <hashar>	 addshore: sorry crashed:  bundle update && bundle exec rake
[14:09:26] <hashar>	 addshore: well apt-cacher-ng I guess we will provide it on the build host if any
[14:12:41] <addshore>	 hmm, let me try a total rebuild / get rid of the buold cache
[14:13:30] <hashar>	 ho man
[14:13:30] <hashar>	 build.sh
[14:14:19] <addshore>	 we could be able to pass --no-cache into build.sh
[14:15:56] <hashar>	 yeah 
[14:16:04] <hashar>	 I was about to write a patch for that :]
[14:16:25] <addshore>	 hashar: do it! :D
[14:16:31] <addshore>	 and maybe one to fix the tabs vs spaces? ;)
[14:16:38] <hashar>	 then I guess I will rewrite it to python :]
[14:16:43] <hashar>	 arg / options handling is no fun in bash
[14:17:17] <addshore>	 indeed
[14:18:35] <addshore>	 amusingly if you write it in python, im gonna write a docker run command to actually run it for me ;)
[14:24:17] <addshore>	 hashar: even after a full rebuild still get the same bundler take gemnotfound issue, im guessing it has somehting do do with running it as "nobody" :/
[14:24:18] <addshore>	 ffs
[14:24:34] <hashar>	 :(
[14:24:44] <hashar>	 I guess bundler is unable to install the gems
[14:25:42] <wmf-insecte>	 Project selenium-MobileFrontend » firefox,beta,Linux,BrowserTests build #569: 04FAILURE in 1 min 57 sec: https://integration.wikimedia.org/ci/job/selenium-MobileFrontend/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=BrowserTests/569/
[14:28:03] <addshore>	 https://www.irccloud.com/pastebin/l6opi5cc/
[14:28:33] <addshore>	 maybe it shouldnt be run as root in the first place? hmmm, thats one of the ones you missed when you went around fixing the dont run as root issues!
[14:32:46] <addshore>	 I dont think this image needs to be multi stage now either, but meh
[14:47:22] <wmf-insecte>	 Yippee, build fixed!
[14:47:22] <wmf-insecte>	 Project selenium-MobileFrontend » firefox,beta,Linux,BrowserTests build #570: 09FIXED in 6 min 12 sec: https://integration.wikimedia.org/ci/job/selenium-MobileFrontend/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=BrowserTests/570/
[14:49:38] <wikibugs_>	 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team, 10MediaWiki-Parser, 10Readers-Web-Backlog (Tracking): Templates rendering as links on beta cluster - https://phabricator.wikimedia.org/T173576#3627374 (10Jdlrobson) Dear Release engineering team, do we know how to increase this size? If we are purp...
[15:04:57] <wmf-insecte>	 Project mediawiki-core-code-coverage build #3023: 04FAILURE in 4 min 56 sec: https://integration.wikimedia.org/ci/job/mediawiki-core-code-coverage/3023/
[15:05:14] <wmf-insecte>	 Project beta-scap-eqiad build #174188: 04FAILURE in 1 min 29 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/174188/
[15:18:49] <wikibugs_>	 (03PS1) 10Hashar: dockerfile: port build script to python [integration/config] - 10https://gerrit.wikimedia.org/r/379775
[15:18:55] <hashar>	 addshore: python!!! ^^
[15:19:23] <wikibugs_>	 (03CR) 10Hashar: "Very rough :]" [integration/config] - 10https://gerrit.wikimedia.org/r/379775 (owner: 10Hashar)
[15:19:47] <wikibugs_>	 (03CR) 10Hashar: [C: 04-1] "Should be ported to the python version https://gerrit.wikimedia.org/r/#/c/379775/" [integration/config] - 10https://gerrit.wikimedia.org/r/379507 (owner: 10Hashar)
[15:21:25] <wikibugs_>	 (03PS1) 10Umherirrender: [BlueSpiceExtendedFilelist] Make unit test voting [integration/config] - 10https://gerrit.wikimedia.org/r/379776
[15:24:06] <hashar>	 !log Restarted Jenkins (out of memory)
[15:24:10] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[15:27:19] <wmf-insecte>	 Yippee, build fixed!
[15:27:19] <wmf-insecte>	 Project beta-scap-eqiad build #174189: 09FIXED in 2 min 36 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/174189/
[15:49:11] <wikibugs_>	 10Release-Engineering-Team, 10Page-Previews, 10Performance-Team (Radar), 10Readers-Web-Backlog (Tracking), 10User-zeljkofilipin: Provide a reliable test environment that mimics production for running integration tests - https://phabricator.wikimedia.org/T174786#3627552 (10Jdlrobson)
[15:57:47] <wikibugs_>	 (03CR) 10Thcipriani: "I knew it would happen eventually (and it's good that it is happening) :)" (037 comments) [integration/config] - 10https://gerrit.wikimedia.org/r/379775 (owner: 10Hashar)
[16:36:08] <addshore>	 hasharAway: ooo, ill look! :D
[16:52:54] <shinken-wm>	 PROBLEM - Puppet errors on deployment-mira is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0]
[17:22:57] <addshore>	 !log docker push docker.io/wmfreleng/tox:v2017.09.22.17.16 & latest # (From current master)
[17:23:00] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[17:25:10] <addshore>	 legoktm: whats the deal with different tox versions?
[17:29:07] <wikibugs_>	 10Beta-Cluster-Infrastructure, 10Continuous-Integration-Infrastructure, 10Operations-Software-Development, 10Technical-Debt: Replace salt on integration and deployment-prep projects - https://phabricator.wikimedia.org/T176314#3627900 (10hashar) I have filled this task for what it is: replace salt on integr...
[17:32:53] <shinken-wm>	 RECOVERY - Puppet errors on deployment-mira is OK: OK: Less than 1.00% above the threshold [0.0]
[17:36:12] <paladox>	 draft support in gerrit upstream being removed next week :)
[17:36:36] <paladox>	 all new changes are created as wip so we will need to get the bot nearer to the time to ignore wip changes
[17:36:44] <paladox>	 otherwise there will be alot of noise
[17:43:00] <mutante>	 paladox: better move your drafts somewhere else now :)
[17:43:06] <paladox>	 lol
[17:43:21] <paladox>	 there's a migration script included so when you run init, it will do it for us
[17:43:28] <mutante>	 heh, cool
[17:43:33] <paladox>	 though the person doing it has to choose weather they are wip or private changes
[17:44:27] <mutante>	 all the drafts were public before, so they should be WIP .. afaict
[17:44:56] <mutante>	 draft and WIP is just another word for the same thing?
[17:45:26] <paladox>	 drafts were hidden from view, but you could still git clone it if you knew the url, privates is complete private
[17:45:37] <paladox>	 wip means work in progress
[17:46:22] <mutante>	 and you could still open the url in browser too, right
[17:46:36] <mutante>	 hidden from view = not in the global list ?
[17:46:59] <mutante>	 or does it even just mean "no notifications by IRC bot"?
[17:47:17] <paladox>	 you can open the url, but if you doint have access to it, it will throw the error emojie
[17:47:49] <mutante>	 hmm, but you can still git clone it at the same time.. that's a weird combo, yea
[17:47:59] <paladox>	 there's no more new events so we will need to check the private or wip propeties in the changes rest api.
[17:48:28] * paladox added support for alot of missing change actions in polygerrit.
[17:49:04] <mutante>	 i wonder how many drafts there are currently
[17:49:23] <paladox>	 heh
[17:49:29] <mutante>	 and percentage of users using that feature
[17:49:44] <paladox>	 everyone that creates a change through the webui will have used drafts
[17:49:48] <mutante>	 and of that, how many are yours ?:)
[17:49:55] <paladox>	 i have probaly 5+
[17:50:06] <paladox>	 mostly things for me to use locally on a instance
[17:50:23] <paladox>	 that reverts some incompatible things in puppet for gerrit
[17:50:41] <mutante>	 hmm, while i have edited exisiting changes through the webui occasionally, i never created them from scratch starting with webeditor
[17:50:59] <paladox>	 editing changes uses change edit
[17:51:13] <paladox>	 no one can view other users change edit until they publish them
[17:51:45] <paladox>	 i think polygerrit will become the default ui in 2.15 if inline edit gets supported.
[17:52:41] <paladox>	 i also made sure that they are going to fix caching issues due to the use of html imports (created an issue). (which i made a blocker of 2.15, though they may disregard it, i hope they doint)
[17:57:19] <wikibugs_>	 10Continuous-Integration-Config, 10VisualEditor, 10VisualEditor-ContentEditable, 10Patch-For-Review: VisualEditor IME test failing in CI on pull-through, but not locally or in the source repo? - https://phabricator.wikimedia.org/T176453#3627932 (10Jdforrester-WMF) OK, it's actually failing all IME tests; t...
[18:56:11] <wikibugs_>	 (03PS2) 10Addshore: docker: lintr image - linter for R [integration/config] - 10https://gerrit.wikimedia.org/r/378831 (https://phabricator.wikimedia.org/T176194)
[19:07:16] <wikibugs_>	 (03PS1) 10Addshore: Add experimental lintr job for analytics/wmde/WDCM [integration/config] - 10https://gerrit.wikimedia.org/r/379818 (https://phabricator.wikimedia.org/T176194)
[19:19:10] <Sagan>	 does somebody now the new link to the gerrit review queue tool (if there is one)? formalry korma.wmflabs.org
[19:19:55] <mutante>	 Sagan: https://www.mediawiki.org/wiki/Community_metrics#wikimedia.biterg.io ?
[19:20:03] <mutante>	 https://wikimedia.biterg.io/app/kibana#/dashboard/Overview
[19:21:10] <Sagan>	 can I search there for old patches and inactive repos too?
[19:22:03] <mutante>	 i dont know that part.. 
[19:22:09] <mutante>	 paladox: might know
[19:22:26] <paladox>	 old patches and inactive repos?
[19:22:44] * paladox dosen't know either, doint use that site alot
[19:23:17] <mutante>	 yea, let's define  inactive repo 
[19:23:38] <Sagan>	 inactive for a year or something like that?
[19:24:20] <mutante>	 i dont think either gerrit or that bitergia software has that kind of categorization into active/inactive
[19:24:56] <mutante>	 so it should just be there if it has (any) history
[19:24:57] <Sagan>	 or inactive for a different period?
[19:25:21] <Sagan>	 before this was: "Reason: Top item on http://korma.wmflabs.org/browser/gerrit_review_queue.html"
[19:26:13] <mutante>	 ah.. maybe Quim knows that
[19:26:36] <wikibugs_>	 (03PS1) 10Hashar: dib: drop aptly repository for php5.5 [integration/config] - 10https://gerrit.wikimedia.org/r/379824 (https://phabricator.wikimedia.org/T174972)
[19:26:42] <mutante>	 maybe ask on a ticket what replaces that specific Korma feature
[19:27:05] <Sagan>	 ok :)
[19:28:26] <Sagan>	 T176514 now
[19:28:27] <stashbot>	 T176514: Find inactive gerrit repos - https://phabricator.wikimedia.org/T176514
[19:29:24] <mutante>	 :)
[19:39:13] <hasharAway>	 I love people filling tasks
[19:39:20] <hasharAway>	 so much that I feel obligated to reply :]
[19:39:56] <wikibugs_>	 10Release-Engineering-Team (Backlog), 10Cleanup, 10Repository-Admins, 10Patch-For-Review, 10User-MarcoAurelio: Deprecate unmaintained/inactive WikiTwidget extension - https://phabricator.wikimedia.org/T148826#3628172 (10MarcoAurelio)
[19:40:56] <mutante>	 hasharAway: :))
[19:41:22] <mutante>	 Sagan: saw it, there's your answer 
[19:42:55] <Sagan>	 hasharAway: do you know a way to find inactive repos?
[19:47:26] <mutante>	 Sagan: what do you want to do with them once you find them?
[19:48:19] <mutante>	 maybe describe the higher level goal and somebody can get a list from shell
[19:48:21] <Sagan>	 mutante: before some time people started to clean up inactive repos, archiving them, so I'd like to take a look there are new ones
[19:48:54] <mutante>	 what does clean up/archiving mean specifically
[19:49:00] <mutante>	 not deleting any files, right?
[19:49:06] <Sagan>	 see https://www.mediawiki.org/wiki/Gerrit/Inactive_projects
[19:49:08] <Sagan>	 archiving etc
[19:49:20] <Sagan>	 marking as unmaintained, put the repo to read-only
[19:49:22] <Sagan>	 no deleting, yep
[19:51:01] <mutante>	 though it does mention "Remove all files in a new commit"  i am not sure i agree that is a good thing
[19:51:08] <mutante>	 depends on each case i guess
[19:51:58] <mutante>	 more often than not i think it doesn't get a real advantage to mark it as "possibly inactive".. it either is or it's not, one day somebody might show up and submit a patch
[19:52:16] <mutante>	 maybe we could look instead if it ever gets cloned
[19:52:30] <mutante>	 and if nobody ever clones it that is the real sign of inactivity
[19:53:09] <mutante>	 i am kind of an inclusionist, so if in doubt i would always just keep it as it is and say disk space isnt our problem
[19:53:39] <Sagan>	 "say disk space isnt our problem" <-- clearing a repo does not make it smaller ;)
[19:53:47] <Sagan>	 only deleting, but that's not the part of the process
[19:53:59] <mutante>	 sometimes i google stuff and then find messages like "archived" or somebody said on a forum thread "you know this is years old, right, archived" but it didnt matter, i was STILL searching for it now and wanted to know
[19:54:02] <Sagan>	 maybe we can change the process to just mark it as inactive
[19:54:53] <mutante>	 Sagan: yea, it doesnt even make it smaller
[19:55:13] <mutante>	 Sagan: what if it just says "last update was X days ago" at the top, very prominently
[19:55:47] <mutante>	 then you dont even need to follow and revert that manual process if it becomes active again
[19:57:21] <mutante>	 Sagan: different color in web ui, if inactive, make everything look grey as a warning :)
[19:57:42] <mutante>	 but automatically
[20:03:41] <hasharAway>	 !log Updating nodepool image for jessie
[20:03:41] <Sagan>	 sounds like a good idea
[20:03:45] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[20:04:13] <hashar>	 Sagan: that depends on the definition of "inactive" repo
[20:04:17] <hashar>	 but mostly we dont really care :]
[20:04:24] <hashar>	 the code just remains around :]
[20:04:49] <hashar>	 though for mediawiki/extensions some are marked read-only in Gerrit / updated on mediawiki.org to state they are gone etc
[20:16:08] <wikibugs_>	 (03PS1) 10Hashar: nodepool: force upgrade packages in snapshots [integration/config] - 10https://gerrit.wikimedia.org/r/379839
[20:28:10] <hashar>	 !log updating nodepool image for jessie [2/x]
[20:28:14] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[20:36:30] <wikibugs_>	 10Scap: scap sync failed on i18n - https://phabricator.wikimedia.org/T175041#3628314 (10thcipriani) 05Open>03Resolved
[20:37:53] <hashar>	 !log Image snapshot-ci-jessie-1506112074 in wmflabs-eqiad is ready
[20:37:57] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[20:52:13] <wikibugs_>	 10Release-Engineering-Team (Backlog), 10Cleanup, 10Repository-Admins, 10Patch-For-Review, 10User-MarcoAurelio: Deprecate unmaintained/inactive WikiTwidget extension - https://phabricator.wikimedia.org/T148826#2734096 (10greg) (@MarcoAurelio in the future, please just add tasks to #releng, not #releng-bac...
[20:52:28] <wikibugs_>	 10Release-Engineering-Team (Backlog), 10Cleanup, 10Repository-Admins, 10Patch-For-Review, 10User-MarcoAurelio: Deprecate unmaintained/inactive WikiTwidget extension - https://phabricator.wikimedia.org/T148826#3628382 (10greg)
[20:55:08] <wikibugs_>	 (03PS1) 10Hashar: dib: php5.5 packages are now in contint::packages::php [integration/config] - 10https://gerrit.wikimedia.org/r/379875 (https://phabricator.wikimedia.org/T174972)
[20:57:35] <wikibugs_>	 10Release-Engineering-Team (Kanban), 10Phabricator, 10User-greg: Form 33 amendment request - https://phabricator.wikimedia.org/T176516#3628418 (10greg)
[21:00:20] <tabbycat>	 thanks greg-g 
[21:01:49] <greg-g>	 tabbycat: np!
[21:02:13] <wikibugs_>	 10Continuous-Integration-Infrastructure (phase-out-trusty), 10Release-Engineering-Team (Kanban), 10Patch-For-Review: Package php modules for Zend 5.5 on Jessie - https://phabricator.wikimedia.org/T174972#3628424 (10hashar) The instances provision fine using apt.wikimedia.org.  The last two patches would let...
[21:02:34] <hashar>	 so php5.5 is almost {done}
[21:02:56] <hashar>	 two last patches and that will be definitely completed
[21:04:25] <wikibugs_>	 (03CR) 10GoranSMilovanovic: [C: 031] Add experimental lintr job for analytics/wmde/WDCM [integration/config] - 10https://gerrit.wikimedia.org/r/379818 (https://phabricator.wikimedia.org/T176194) (owner: 10Addshore)
[21:04:49] <wikibugs_>	 (03CR) 10GoranSMilovanovic: [C: 031] docker: lintr image - linter for R [integration/config] - 10https://gerrit.wikimedia.org/r/378831 (https://phabricator.wikimedia.org/T176194) (owner: 10Addshore)
[21:06:35] <addshore>	 hashar: ill try the lintr job tommorrow, bed now
[21:07:37] <icinga-wm>	 PROBLEM - Work requests waiting in Zuul Gearman server https://grafana.wikimedia.org/dashboard/db/zuul-gearman on contint1001 is CRITICAL: CRITICAL: 57.14% of data above the critical threshold [140.0]
[21:13:55] <wmf-insecte>	 Project beta-scap-eqiad build #174223: 04FAILURE in 1 min 45 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/174223/
[21:15:06] <wmf-insecte>	 Project beta-scap-eqiad build #174224: 04STILL FAILING in 28 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/174224/
[21:16:48] <thcipriani>	 ^ looking
[21:17:21] <icinga-wm>	 ACKNOWLEDGEMENT - Work requests waiting in Zuul Gearman server https://grafana.wikimedia.org/dashboard/db/zuul-gearman on contint1001 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [140.0] amusso Transient mass changes on mediawiki/core
[21:19:45] <hashar>	  There are too many unreachable loose objects; run 'git prune' to remove them.
[21:19:45] <hashar>	 hehe
[21:20:01] <hashar>	 thcipriani: 100% sure we have a task about making scap to git gc stuff from time to time
[21:21:40] <thcipriani>	 hashar: so currently scap runs https://github.com/wikimedia/scap/blob/master/scap/git.py#L219 on /srv/mediawiki every run in beta
[21:22:08] <thcipriani>	 but there is still quite a bit of growth on that repo (as we've seen)
[21:22:17] <wikibugs_>	 10Release-Engineering-Team (Kanban), 10User-greg: End of Q1 grooming - https://phabricator.wikimedia.org/T176523#3628523 (10greg)
[21:22:19] <shinken-wm>	 PROBLEM - Puppet errors on aptly is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0]
[21:22:33] <thcipriani>	 (next beta-scap-eqiad should work, FYI)
[21:25:38] <wikibugs_>	 10Release-Engineering-Team (Kanban), 10User-greg: 201718Q2 RelEng related program goals - https://phabricator.wikimedia.org/T174835#3628538 (10greg) 05Open>03Resolved Pretty much done. Any changes (if any) will be minor at this point.
[21:29:04] <wmf-insecte>	 Yippee, build fixed!
[21:29:04] <wmf-insecte>	 Project beta-scap-eqiad build #174225: 09FIXED in 5 min 21 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/174225/
[21:29:08] <hashar>	 thcipriani: \o/
[21:29:33] <hashar>	 and potentially git submodules need to get git gc run on them
[21:29:39] <hashar>	 something like git submodule foreach git gc
[21:29:48] <hashar>	 but hey who knows really :(
[21:29:59] <thcipriani>	 ohh boy
[21:30:35] <hashar>	 it is too late for me to figure out why it errors out despite scap running git gc
[21:38:00] <wikibugs_>	 10Release-Engineering-Team (Backlog), 10User-greg: Make a table of access levels per service RelEng maintains per person - https://phabricator.wikimedia.org/T135187#3628608 (10greg) 05Open>03Resolved a:03hashar I don't know why I didn't close this before. I guess I had something in my mind on how to impr...
[21:38:17] <shinken-wm>	 PROBLEM - Puppet errors on integration-slave-docker-1001 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0]
[21:47:41] <wikibugs_>	 10Release-Engineering-Team, 10Cleanup, 10Repository-Admins, 10User-MarcoAurelio: Deprecate unmaintained/inactive WikiTwidget extension - https://phabricator.wikimedia.org/T148826#3628627 (10MarcoAurelio) 05Open>03Resolved Thanks @greg :)
[21:55:19] <tabbycat>	 !log Granted Greg G. 'staff' global rights on the beta cluster per request
[21:55:23] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[22:06:33] <wikibugs_>	 10Deployment-Systems, 10Release-Engineering-Team, 10Patch-For-Review: It shouldn't be possible to create WMF branches on master - https://phabricator.wikimedia.org/T175324#3628674 (10demon) What I said on IRC after going down a very deep rabbit hole with @thcipriani on this.  ``` 14:56 Easy way to find when...
[22:13:19] <shinken-wm>	 RECOVERY - Puppet errors on integration-slave-docker-1001 is OK: OK: Less than 1.00% above the threshold [0.0]
[22:39:13] <wikibugs_>	 10Gerrit, 10MediaWiki-Vagrant, 10Patch-For-Review: "index-pack failed" when installing new MediaWiki-Vagrant box - https://phabricator.wikimedia.org/T152801#2860596 (10Byrnedj12) I had to make two changes:  1) in puppet/modules/mediawiki/manifests/init.pp,  set http.postBuffer=2098576000 2) in support/Vagran...
[22:39:20] <shinken-wm>	 PROBLEM - Puppet errors on integration-slave-docker-1001 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0]
[22:43:07] <icinga-wm>	 RECOVERY - Work requests waiting in Zuul Gearman server https://grafana.wikimedia.org/dashboard/db/zuul-gearman on contint1001 is OK: OK: Less than 30.00% above the threshold [90.0]
[23:14:17] <shinken-wm>	 RECOVERY - Puppet errors on integration-slave-docker-1001 is OK: OK: Less than 1.00% above the threshold [0.0]
[23:40:18] <shinken-wm>	 PROBLEM - Puppet errors on integration-slave-docker-1001 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0]
[23:49:13] <wikibugs_>	 10Continuous-Integration-Config, 10Gerrit, 10Cleanup, 10Diffusion, 10GitHub-Mirrors: Tool to archive extensions (and do related stuff)? - https://phabricator.wikimedia.org/T175499#3628842 (10Liuxinyu970226) Note: a "template" for requesting archive is provided by @mmodell: T174410. But it would be love t...
[23:57:45] <wikibugs_>	 10Continuous-Integration-Config, 10Gerrit, 10Cleanup, 10Diffusion, 10GitHub-Mirrors: Tool to archive extensions (and do related stuff)? - https://phabricator.wikimedia.org/T175499#3628859 (10demon) >>! In T175499#3628842, @Liuxinyu970226 wrote: > Note: a "template" for requesting archive is provided by @...