[00:01:32] <wikibugs>	 10Beta-Cluster-Infrastructure, 10MediaWiki-extensions-ORES, 10ORES, 06Revision-Scoring-As-A-Service, 07Spike: [Spike] Should we make a model for ores in beta? - https://phabricator.wikimedia.org/T141980#2521305 (10Legoktm) >>! In T141980#2519511, @Halfak wrote: > However, I think it would be fine to ceas...
[00:38:34] <wikibugs>	 06Release-Engineering-Team, 15User-greg, 15User-zeljkofilipin: Add a European mid-day SWAT window - https://phabricator.wikimedia.org/T137970#2385565 (10aude) I can't do swat everyday, but probably can help some of the time with this new timeslot
[02:11:50] <wikibugs>	 10Beta-Cluster-Infrastructure, 06Labs, 10Labs-Infrastructure, 07Tracking: Log files on labs instance fill up disk (/var is only 2GB) (tracking) - https://phabricator.wikimedia.org/T71601#2521463 (10Tgr)
[02:11:53] <wikibugs>	 10Beta-Cluster-Infrastructure: deployment-sentry2 /var is full - https://phabricator.wikimedia.org/T132730#2521460 (10Tgr) 05Open>03Resolved a:03Tgr >>! In T132730#2521199, @greg wrote: > Can sentry2 be deleted then? We're up against our project quota again (trying to rebuild our db servers on jessie).  It...
[03:19:34] <wikibugs>	 07Browser-Tests, 06Reading-Web-Backlog, 03Reading-Web-Sprint-78-Terminal-Velocity, 07Regression, 07Unplanned-Sprint-Work: [Regression] Fix browser tests for language switching on the beta cluster - https://phabricator.wikimedia.org/T141647#2521476 (10phuedx) 05Open>03stalled Per T141647#2520376
[03:48:44] <wikibugs>	 10Beta-Cluster-Infrastructure: deployment-sentry2 /var is full - https://phabricator.wikimedia.org/T132730#2521497 (10greg) Thanks, sorry for the noise and me not double checking. I was going off of an old list.
[05:54:27] <Amir1>	 !log deploying 616707c to ores
[06:01:57] <Amir1>	 !log ran puppet agent in deployment-sca03 
[06:03:54] <Amir1>	 !log ran puppet agent in deployment-ores-redis 
[06:04:20] <Amir1>	 !log restarting redis-instance-tcp_6379, redis-instance-tcp_6380, and redis-server services in deployment-ores-redis
[06:05:11] <Amir1>	 !log restarting uwsgi-ores and celery-ores-worker in deployment-sca03
[06:58:15] <shinken-wm>	 PROBLEM - Puppet run on phab-beta is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0]
[08:05:10] <shinken-wm>	 PROBLEM - English Wikipedia Main page on beta-cluster is CRITICAL: HTTP CRITICAL: HTTP/1.1 301 TLS Redirect - string 'Wikipedia' not found on 'http://en.wikipedia.beta.wmflabs.org:80/wiki/Main_Page?debug=true' - 567 bytes in 0.003 second response time
[08:06:18] <shinken-wm>	 PROBLEM - English Wikipedia Mobile Main page on beta-cluster is CRITICAL: HTTP CRITICAL: HTTP/1.1 301 TLS Redirect - string 'Wikipedia' not found on 'http://en.m.wikipedia.beta.wmflabs.org:80/wiki/Main_Page?debug=true' - 569 bytes in 0.005 second response time
[11:50:30] <shinken-wm>	 PROBLEM - Puppet run on deployment-cache-text04 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0]
[12:05:28] <shinken-wm>	 PROBLEM - Puppet run on deployment-tmh01 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0]
[12:05:37] <wikibugs>	 06Release-Engineering-Team, 07Documentation: Merge Wikimedia's "Deployment checklist for new extensions" doc pages - https://phabricator.wikimedia.org/T142081#2522000 (10Aklapper)
[12:06:23] <paladox>	 Amir1 hi it seems it didnt log
[12:07:45] <Amir1>	 hmm, Others should check
[12:07:50] <paladox>	 Yep
[12:08:37] <paladox>	 Seems log bot is broken in here i can ask the people who are on the ops list but they are all on american time so it is still very earley.
[12:20:05] <paladox>	 Amir1 https://phabricator.wikimedia.org/T141683#2522046
[12:21:12] <Amir1>	 I will check it later, I need to make a presentation for tomorrow morning
[12:22:04] <paladox>	 Ok
[12:22:08] <paladox>	 thanks
[12:40:29] <shinken-wm>	 RECOVERY - Puppet run on deployment-tmh01 is OK: OK: Less than 1.00% above the threshold [0.0]
[14:44:43] <shinken-wm>	 PROBLEM - SSH on deployment-changeprop is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[15:10:07] <paladox>	 Hi, it seems the logs bot in here is not working
[15:13:29] <paladox>	 It seems that qa-morebots hasent rejoined this channel
[16:09:33] <shinken-wm>	 RECOVERY - SSH on deployment-changeprop is OK: SSH OK - OpenSSH_6.7p1 Debian-5+deb8u3 (protocol 2.0)
[16:20:17] <wikibugs>	 10Beta-Cluster-Infrastructure, 10ChangeProp, 10Citoid, 10ContentTranslation-CXserver, 10VisualEditor: Merge service-specific instances in deployment-prep to the deployment-sca* instances - https://phabricator.wikimedia.org/T142100#2522715 (10greg)
[16:21:48] <shinken-wm>	 PROBLEM - Puppet run on deployment-changeprop is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0]
[16:22:47] <wikibugs>	 10Beta-Cluster-Infrastructure, 10ChangeProp, 10Citoid, 10ContentTranslation-CXserver, 10VisualEditor: Merge service-specific instances in deployment-prep to the deployment-sca* instances - https://phabricator.wikimedia.org/T142100#2522744 (10greg)
[16:24:03] <paladox>	 greg-g Hi it seems that the log bout has gone from this channel.
[16:24:12] <paladox>	 I doint see it in the channel list qa-morebots
[16:24:38] <greg-g>	 paladox: ask in -labs
[16:25:10] <paladox>	 I did they said to ask the maintainers
[16:29:44] <shinken-wm>	 PROBLEM - Puppet run on integration-slave-jessie-1003 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0]
[16:30:10] <shinken-wm>	 PROBLEM - Puppet run on integration-slave-trusty-1023 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0]
[16:30:16] <shinken-wm>	 PROBLEM - Puppet run on integration-puppetmaster is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0]
[16:32:21] <shinken-wm>	 PROBLEM - Puppet run on integration-slave-trusty-1006 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0]
[16:32:35] <shinken-wm>	 PROBLEM - Puppet run on integration-publisher is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0]
[16:37:23] <shinken-wm>	 PROBLEM - Puppet run on integration-saltmaster is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0]
[16:40:07] <shinken-wm>	 PROBLEM - Puppet run on integration-slave-trusty-1018 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0]
[16:41:13] <shinken-wm>	 PROBLEM - Puppet run on integration-jessie-lego-test01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0]
[16:41:14] <wikibugs>	 10Beta-Cluster-Infrastructure, 10ChangeProp, 10Citoid, 10ContentTranslation-CXserver, 10VisualEditor: Merge service-specific instances in deployment-prep to the deployment-sca* instances - https://phabricator.wikimedia.org/T142100#2522715 (10mobrovac) - Zotero can't run on Jessie, so can't be moved. - Id...
[16:41:14] <paladox>	 greg-g yay we managed to get gerrit started weave identified what needs improving to get puppet to work. Since we had to use a mix of manual commands but we made a list.
[16:41:15] <paladox>	 https://phabricator.wikimedia.org/P3637
[16:41:39] <paladox>	 We also need to make https optional for labs since theres a limit on letsencrypt
[16:42:07] <shinken-wm>	 PROBLEM - Puppet run on integration-slave-trusty-1011 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0]
[16:42:49] <shinken-wm>	 PROBLEM - Puppet run on integration-slave-trusty-1014 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0]
[16:43:40] <paladox>	 I am going to work on https, and once merged will go through list again to make sure i put every thing in the correct order
[16:44:02] <shinken-wm>	 PROBLEM - Puppet run on integration-slave-jessie-1001 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0]
[16:46:10] <shinken-wm>	 PROBLEM - Puppet run on integration-aptly01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0]
[16:46:36] <shinken-wm>	 PROBLEM - Puppet run on zuul-dev-jessie is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0]
[16:47:52] <shinken-wm>	 PROBLEM - Puppet run on integration-slave-jessie-1002 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0]
[16:47:55] <paladox>	 It seems puppet is failing ^^
[16:48:32] <shinken-wm>	 PROBLEM - Puppet run on integration-raita is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0]
[16:50:01] <thcipriani>	 Connection refused - connect(2) for "integration-puppetmaster.integration.eqiad.wmflabs" port 8140 huh.
[16:50:30] <shinken-wm>	 PROBLEM - Puppet run on integration-slave-trusty-1003 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0]
[16:50:41] <thcipriani>	 sure enough: integration-puppetmaster.integration.eqiad.wmflabs [10.68.16.42] 8140 (?) : Connection refused
[16:52:22] <shinken-wm>	 PROBLEM - Puppet run on castor is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0]
[16:53:37] <thcipriani>	 !log restart integration-puppetmaster puppetmaster service. Evidently OOM'd.
[16:54:03] <thcipriani>	 integration *should* start recovering...
[16:54:33] <shinken-wm>	 PROBLEM - Puppet run on integration-slave-trusty-1001 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0]
[16:55:15] <shinken-wm>	 PROBLEM - Puppet run on integration-slave-trusty-1016 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0]
[16:56:45] <shinken-wm>	 PROBLEM - Puppet run on integration-slave-jessie-android is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0]
[16:57:13] <shinken-wm>	 PROBLEM - Puppet run on integration-slave-trusty-1012 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0]
[16:57:55] <shinken-wm>	 PROBLEM - Puppet run on integration-slave-trusty-1017 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0]
[16:58:24] <greg-g>	 paladox: good deal re gerrit!
[16:58:40] <greg-g>	 paladox: re qa-morebots... the maintainers are listed somewhere, I'm thinking it's legok.tm 
[16:59:16] <legoktm>	 still pinged
[16:59:25] <legoktm>	 I'm totally not the maintainer
[16:59:27] <legoktm>	 :P
[16:59:49] <shinken-wm>	 PROBLEM - Puppet run on integration-slave-trusty-1004 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0]
[16:59:50] <greg-g>	 legoktm: heh, oh, probably on "lego" then?
[17:00:00] <legoktm>	 yep :P
[17:00:04] <greg-g>	 https://wikitech.wikimedia.org/wiki/Morebots you're name is here with other people's!
[17:00:19] <greg-g>	 well, as authhors
[17:00:21] <greg-g>	 -h
[17:03:29] <legoktm>	 !log jstart -N qamorebots /usr/lib/adminbot/adminlogbot.py --config ./confs/qa-logbot.py
[17:03:33] <legoktm>	 !log jstart -N qamorebots /usr/lib/adminbot/adminlogbot.py --config ./confs/qa-logbot.py
[17:03:36] <qa-morebots>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master
[17:03:39] <legoktm>	 :D
[17:04:08] <legoktm>	 greg-g: yeah, I wrote a patch for it a loooong time ago :P
[17:05:15] <wikibugs>	 06Release-Engineering-Team, 06Operations, 15User-greg, 07Wikimedia-Incident: Institute quarterly(?) review of incident reports and follow-up - https://phabricator.wikimedia.org/T141287#2522931 (10greg) Update:  We (Faidon, Kevin S, and myself) just had a conversation about this. Notes at https://etherpad.w...
[17:05:16] <shinken-wm>	 RECOVERY - Puppet run on integration-puppetmaster is OK: OK: Less than 1.00% above the threshold [0.0]
[17:05:26] <greg-g>	 legoktm: <3 <3
[17:07:01] <legoktm>	 :)
[17:07:22] <shinken-wm>	 RECOVERY - Puppet run on integration-slave-trusty-1006 is OK: OK: Less than 1.00% above the threshold [0.0]
[17:07:32] <shinken-wm>	 RECOVERY - Puppet run on integration-publisher is OK: OK: Less than 1.00% above the threshold [0.0]
[17:09:46] <shinken-wm>	 RECOVERY - Puppet run on integration-slave-jessie-1003 is OK: OK: Less than 1.00% above the threshold [0.0]
[17:10:12] <shinken-wm>	 RECOVERY - Puppet run on integration-slave-trusty-1023 is OK: OK: Less than 1.00% above the threshold [0.0]
[17:11:03] <wikibugs>	 10Beta-Cluster-Infrastructure, 06Commons, 10MediaWiki-Authentication-and-authorization, 10MediaWiki-extensions-CentralAuth, 06Reading-Web-Backlog: Unable to log in on https://commons.m.wikimedia.beta.wmflabs.org/wiki/Special:UserLogin - https://phabricator.wikimedia.org/T142015#2522965 (10Jdlrobson) Mobi...
[17:11:07] <wikibugs>	 10Beta-Cluster-Infrastructure, 06Commons, 10MediaWiki-Authentication-and-authorization, 10MediaWiki-extensions-CentralAuth, 06Reading-Web-Backlog: Unable to log in on https://commons.m.wikimedia.beta.wmflabs.org/wiki/Special:UserLogin - https://phabricator.wikimedia.org/T142015#2522967 (10Jdlrobson) Mobi...
[17:11:10] <shinken-wm>	 RECOVERY - Puppet run on integration-jessie-lego-test01 is OK: OK: Less than 1.00% above the threshold [0.0]
[17:12:24] <shinken-wm>	 RECOVERY - Puppet run on integration-saltmaster is OK: OK: Less than 1.00% above the threshold [0.0]
[17:12:50] <shinken-wm>	 RECOVERY - Puppet run on integration-slave-trusty-1014 is OK: OK: Less than 1.00% above the threshold [0.0]
[17:15:09] <shinken-wm>	 RECOVERY - Puppet run on integration-slave-trusty-1018 is OK: OK: Less than 1.00% above the threshold [0.0]
[17:17:11] <shinken-wm>	 RECOVERY - Puppet run on integration-slave-trusty-1011 is OK: OK: Less than 1.00% above the threshold [0.0]
[17:19:30] <paladox>	 greg-g sorry for late reply yeh, I did some commands manually as the list says but only to find out what fixes we need to do
[17:19:47] <paladox>	 it seems that gerrit2 permission on some files gets wiped too
[17:19:48] <wikibugs>	 06Release-Engineering-Team, 15User-greg, 15User-zeljkofilipin: Add a European mid-day SWAT window - https://phabricator.wikimedia.org/T137970#2523037 (10greg) >>! In T137970#2521390, @aude wrote: > I can't do swat everyday, but probably can help some of the time with this new timeslot  so to be explicit, sho...
[17:19:56] <paladox>	 legoktm thanks
[17:20:19] <jdlrobson>	 thcipriani: hey! Any chance you could take a look at https://gerrit.wikimedia.org/r/302775 ? Reading's browser tests are all failing which is making it tricky for us to catch regressions. I think this should take care of it based on what you've told me.
[17:21:09] <shinken-wm>	 RECOVERY - Puppet run on integration-aptly01 is OK: OK: Less than 1.00% above the threshold [0.0]
[17:22:30] <thcipriani>	 jdlrobson: ack, sure. Give me a few minutes, watching some scap3 deployment stuff at the moment. I'm honestly not sure where it gets that URL from. I can try your patch/dig a little deeper here in a few.
[17:22:32] <wikibugs>	 06Release-Engineering-Team, 07Documentation, 07Easy: Merge Wikimedia's "Deployment checklist for new extensions" doc pages - https://phabricator.wikimedia.org/T142081#2523049 (10greg) I'm totally fine with mw:Writing_an_extension_for_deployment over mw:Review_queue. "Review queue" isn't immediately obvious w...
[17:23:11] <wikibugs>	 06Release-Engineering-Team, 07Documentation, 07Easy: Merge Wikimedia's "Deployment checklist for new extensions" doc pages - https://phabricator.wikimedia.org/T142081#2523055 (10greg) ...to be explicit: after merge we can probably just redirect "Review queue" to "Writing and extension..."
[17:23:33] <shinken-wm>	 RECOVERY - Puppet run on integration-raita is OK: OK: Less than 1.00% above the threshold [0.0]
[17:24:01] <shinken-wm>	 RECOVERY - Puppet run on integration-slave-jessie-1001 is OK: OK: Less than 1.00% above the threshold [0.0]
[17:24:26] <wikibugs>	 06Release-Engineering-Team, 15User-greg, 15User-zeljkofilipin: Add a European mid-day SWAT window - https://phabricator.wikimedia.org/T137970#2523056 (10aude) >>! In T137970#2523037, @greg wrote: >>>! In T137970#2521390, @aude wrote: >> I can't do swat everyday, but probably can help some of the time with th...
[17:25:31] <shinken-wm>	 RECOVERY - Puppet run on integration-slave-trusty-1003 is OK: OK: Less than 1.00% above the threshold [0.0]
[17:26:34] <shinken-wm>	 RECOVERY - Puppet run on zuul-dev-jessie is OK: OK: Less than 1.00% above the threshold [0.0]
[17:26:35] <jdlrobson>	 thcipriani: the MEDIAWIKI_API is only set there as far as i know - at least for reading's extensions
[17:27:03] <jdlrobson>	 we can override it but that will be a little messy as we'd have to touch a few more extensions (although we might want to try it out on just one extension to test the assumption).
[17:27:24] <shinken-wm>	 RECOVERY - Puppet run on castor is OK: OK: Less than 1.00% above the threshold [0.0]
[17:27:52] <shinken-wm>	 RECOVERY - Puppet run on integration-slave-jessie-1002 is OK: OK: Less than 1.00% above the threshold [0.0]
[17:28:14] <wikibugs>	 06Release-Engineering-Team, 15User-greg, 15User-zeljkofilipin: Add a European mid-day SWAT window - https://phabricator.wikimedia.org/T137970#2523077 (10greg) >>! In T137970#2523056, @aude wrote: > Sure, please add me.  {{done}} thanks!
[17:29:30] <shinken-wm>	 RECOVERY - Puppet run on integration-slave-trusty-1001 is OK: OK: Less than 1.00% above the threshold [0.0]
[17:31:46] <shinken-wm>	 RECOVERY - Puppet run on integration-slave-jessie-android is OK: OK: Less than 1.00% above the threshold [0.0]
[17:33:11] <paladox>	 Yay i get the maintainance page http://gerrit-test3.wmflabs.org/error.html#
[17:34:29] <paladox>	 Ami1 hi, the logs bot is now working in -releng
[17:34:52] <shinken-wm>	 RECOVERY - Puppet run on integration-slave-trusty-1004 is OK: OK: Less than 1.00% above the threshold [0.0]
[17:38:30] <wikibugs>	 06Release-Engineering-Team, 15User-greg, 15User-zeljkofilipin: Add a European mid-day SWAT window - https://phabricator.wikimedia.org/T137970#2523104 (10greg)
[17:40:34] <thcipriani>	 jdlrobson: made you change on the selenium-MobileFrontend if you're fine with it, we could try re-running and see what happens?
[17:46:54] <jdlrobson>	 thcipriani: not sure i follow
[17:47:10] <jdlrobson>	 where did you make the change?
[17:47:40] <thcipriani>	 jdlrobson: I made the modification in your patch on https://integration.wikimedia.org/ci/job/selenium-MobileFrontend/
[17:48:00] <thcipriani>	 this change: https://gerrit.wikimedia.org/r/#/c/302775/1/jjb/job-templates-selenium.yaml
[17:48:33] <jdlrobson>	 thcipriani: so it's live?
[17:48:36] <thcipriani>	 yup
[17:48:39] <jdlrobson>	 ok ill rerun
[17:48:42] <jdlrobson>	 and see what happens
[17:48:44] <thcipriani>	 cool
[17:48:50] <jdlrobson>	 fingers crossed
[17:48:57] <thcipriani>	 ^ +1
[17:50:23] <thcipriani>	 blerg. Looks like the same deal. https://integration.wikimedia.org/ci/job/selenium-MobileFrontend/107/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=contintLabsSlave%20&&%20UbuntuTrusty/console
[17:51:56] <thcipriani>	 ok. I'll dig a bit. I saw a flurry of patches for various gems last night. I'll see if I can catch up.
[18:00:12] <shinken-wm>	 RECOVERY - Puppet run on integration-slave-trusty-1016 is OK: OK: Less than 1.00% above the threshold [0.0]
[18:02:13] <shinken-wm>	 RECOVERY - Puppet run on integration-slave-trusty-1012 is OK: OK: Less than 1.00% above the threshold [0.0]
[18:02:53] <shinken-wm>	 RECOVERY - Puppet run on integration-slave-trusty-1017 is OK: OK: Less than 1.00% above the threshold [0.0]
[18:06:57] <jdlrobson>	 thcipriani: api step still failing with same error :(
[18:07:15] <thcipriani>	 indeed.
[18:07:49] <jdlrobson>	 thcipriani:anyway we can verify the API variable did get set correctly?
[18:31:41] <wikibugs>	 06Release-Engineering-Team (Deployment-Blockers), 05Release: MW-1.28.0-wmf.17 deployment blockers - https://phabricator.wikimedia.org/T142117#2523354 (10greg)
[18:32:04] <wikibugs>	 06Release-Engineering-Team (Deployment-Blockers), 05Release: MW-1.28.0-wmf.16 deployment blockers - https://phabricator.wikimedia.org/T141551#2502733 (10greg)
[18:36:08] <wikibugs>	 06Release-Engineering-Team (Deployment-Blockers), 05Release: MW-1.28.0-wmf.17 deployment blockers - https://phabricator.wikimedia.org/T142117#2523393 (10greg) a:03hashar
[18:40:14] <wikibugs>	 10Beta-Cluster-Infrastructure, 10MediaWiki-extensions-ORES, 06Revision-Scoring-As-A-Service: Dashboard or pane for ORES failed jobs on beta - https://phabricator.wikimedia.org/T142119#2523395 (10Ladsgroup)
[18:47:33] <tgr>	 do we have coverage maps for the core unit tests?
[18:48:51] <thcipriani>	 tgr: we have this https://doc.wikimedia.org/cover/mediawiki-core/master/php/
[18:50:03] <tgr>	 thanks, just what I needed
[18:51:11] <tgr>	 thcipriani: the unit test for https://gerrit.wikimedia.org/r/#/c/302386 is segfaulting but there are no tests at all for the changed class (or so I'd assume from the coverage report)
[18:51:16] <tgr>	 is that just random noise?
[18:51:33] * thcipriani looks
[18:52:06] <tgr>	 oh, the segfaults are at different tests
[18:52:13] <tgr>	 just noise then I suppose
[18:54:27] <ostriches>	 thcipriani: Did some rsync optimization. https://phabricator.wikimedia.org/D304
[18:56:18] <thcipriani>	 ostriches: nice, yeah we should clean that up a bit.
[18:56:43] <thcipriani>	 tgr: it might be noise. weird that it happened on two different hosts for the same patch set though.
[18:58:57] <tgr>	 let's try a third time then
[18:59:07] <tgr>	 how do I remove reviews in new gerrit?
[19:00:35] <paladox>	 tgr you click the x
[19:00:40] <paladox>	 on the reviewer
[19:01:33] <paladox>	 Its a small
[19:01:34] <paladox>	 x
[19:02:55] <tgr>	 ah, yeah, you cannot remove reviews without removing reviewers
[19:02:57] <tgr>	 thanks
[19:03:15] <paladox>	 Yep
[19:03:24] <paladox>	 tgr you should if you have the permission in gerrit to
[19:06:36] <paladox>	 ostriches sorry for ping, but im wondering with the gerrit package it seems that some permissions are for root and some for gerrit2
[19:06:41] <paladox>	 which causes failures
[19:07:02] <paladox>	 See my list https://phabricator.wikimedia.org/P3637 on what i have to do to get it working
[19:11:10] <thcipriani>	 tgr: huh, guess it was just noise. I'll keep an eye on the two boxen that segfaulted.
[19:11:15] <ostriches>	  sudo chown -R gerrit2 /var/lib/gerrit2/review_site/etc/ssh_host* sounds like init was run by root and not gerrit2
[19:11:34] <paladox>	 Yep
[19:11:40] <ostriches>	 So yeah not a bug :)
[19:11:43] <paladox>	 Oh
[19:11:46] <ostriches>	 That one, at least.
[19:11:51] <paladox>	 Wait i was meant to run it as gerrit2
[19:11:59] <ostriches>	 Yeah
[19:12:02] <paladox>	 Oh that explains most of my problems lol
[19:12:05] <ostriches>	 You should always run gerrit.war as gerrit2.
[19:12:14] <paladox>	 Oh ah, thanks. Once your patch
[19:12:20] <paladox>	 for gerrit 2.12.2.wmf2
[19:12:27] <ostriches>	 Otherwise root will end up writing files that gerrit2 can't access.
[19:12:28] <paladox>	 is merged that will fix the problem too
[19:12:31] <paladox>	 oh
[19:12:36] <paladox>	 ah, thanks for explaning.
[19:13:06] <paladox>	 Im going to make https optional for now in labs, we can work on that later, but then i will re do my steps changing
[19:13:07] <paladox>	 them
[19:13:14] <paladox>	 thanks for that, didnt know that :)
[19:15:14] <ostriches>	 Yeah no worries. When working on a gerrit box, run things as gerrit2. Except like restarting a service or running puppet, which is root.
[19:15:42] <paladox>	 Ok thanks
[19:15:44] <ostriches>	 Easiest fix in that case is `chown -R gerrit2:gerrit2 /var/lib/gerrit2
[19:15:50] <paladox>	 Oh yep
[19:15:54] <paladox>	 Oh that is easy
[19:15:55] <paladox>	 thanks
[19:16:08] <paladox>	 I will delete the isntance and recreate it soon.
[19:16:15] <paladox>	 It managed to start though.
[19:21:29] * paladox loves it when i close internet explorer and reopen to find that i can restore my tabs
[19:21:43] * paladox needs to upgrade his cpu, 1.8ghz too slow.
[19:37:55] <ostriches>	 paladox: I rewrote https://gerrit.wikimedia.org/r/#/c/302852/ in a simpler way
[19:38:05] <paladox>	 Oh thanksyou
[19:38:08] <paladox>	 thankyou
[19:38:09] <ostriches>	 Give that a shot in labs with gerrit::proxy::lets_encrypt set to false :)
[19:38:17] <paladox>	 Ok, how do i apply it
[19:38:18] <paladox>	 ?
[19:38:21] <ostriches>	 I haven't tested yet, but it should work :)
[19:38:32] <paladox>	 Ok, not sure what command you use 
[19:39:39] <paladox>	 ostriches oh wait that will break
[19:39:43] <paladox>	 since we need proxy pass
[19:39:53] <ostriches>	 You get it still :)
[19:40:07] <paladox>	 Not sure what you mean?
[19:40:18] <ostriches>	 Do you see where the if/end are placed?
[19:40:26] <paladox>	 Yep
[19:40:32] <paladox>	 But wont ssl break things
[19:40:50] <ostriches>	 Oh dammit I tried to be too clever.
[19:40:55] * ostriches rethinks.
[19:40:55] <paladox>	 Oh lol
[19:41:20] <grrrit-wm>	 (03CR) 10Chad: [C: 032] Bump mediawiki_api dependency for redirect support [selenium] - 10https://gerrit.wikimedia.org/r/302849 (owner: 10Dduvall)
[19:43:29] <grrrit-wm>	 (03CR) 10Chad: [C: 032] Release patch version 1.7.2 [selenium] - 10https://gerrit.wikimedia.org/r/302850 (owner: 10Dduvall)
[19:44:32] <grrrit-wm>	 (03Merged) 10jenkins-bot: Bump mediawiki_api dependency for redirect support [selenium] - 10https://gerrit.wikimedia.org/r/302849 (owner: 10Dduvall)
[19:46:19] <grrrit-wm>	 (03Merged) 10jenkins-bot: Release patch version 1.7.2 [selenium] - 10https://gerrit.wikimedia.org/r/302850 (owner: 10Dduvall)
[19:47:33] <paladox>	 ostriches wont that also fail since it will include 
[19:47:33] <paladox>	 NameVirtualHost *:443
[19:47:34] <paladox>	 <VirtualHost *:443>
[19:47:45] <paladox>	 and also that will break https and http
[20:00:35] <grrrit-wm>	 (03CR) 10Dduvall: [C: 032] API should use https on the beta cluster [integration/config] - 10https://gerrit.wikimedia.org/r/302775 (https://phabricator.wikimedia.org/T129483) (owner: 10Jdlrobson)
[20:01:51] <grrrit-wm>	 (03Merged) 10jenkins-bot: API should use https on the beta cluster [integration/config] - 10https://gerrit.wikimedia.org/r/302775 (https://phabricator.wikimedia.org/T129483) (owner: 10Jdlrobson)
[20:05:07] <wikibugs>	 05Gitblit-Deprecate, 10Diffusion: Redirect git.wikimedia.org HEAD URLs to Diffusion - https://phabricator.wikimedia.org/T141965#2523736 (10Danny_B) >>! In T141965#2518195, @Aklapper wrote: > Expected target should be https://phabricator.wikimedia.org/diffusion/GTWN/browse/irc-relay IMO  That's not the one I ha...
[20:05:30] <wikibugs>	 05Gitblit-Deprecate, 10Diffusion: Redirect git.wikimedia.org HEAD URLs to Diffusion - https://phabricator.wikimedia.org/T141965#2523737 (10Danny_B) Is it presumable that HEAD should always link to master?
[20:05:43] <wikibugs>	 06Release-Engineering-Team, 06Operations, 10Traffic, 07HTTPS: Retire gerrit.wikimedia.org SSL cert - https://phabricator.wikimedia.org/T142131#2523742 (10demon)
[20:07:18] <marxarelli>	 !log Running jenkins-jobs update config/ 'selenium-*' to deploy https://gerrit.wikimedia.org/r/#/c/302775/
[20:07:22] <qa-morebots>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master
[20:09:12] <grrrit-wm>	 (03CR) 10Dduvall: "$ jenkins-jobs --conf etc/jenkins_jobs.ini update config/ 'selenium-*'" [integration/config] - 10https://gerrit.wikimedia.org/r/302775 (https://phabricator.wikimedia.org/T129483) (owner: 10Jdlrobson)
[20:09:44] <wikibugs>	 05Gitblit-Deprecate, 10Diffusion: Redirect git.wikimedia.org HEAD URLs to Diffusion - https://phabricator.wikimedia.org/T141965#2518118 (10greg) >>! In T141965#2523737, @Danny_B wrote: > Is it presumable that HEAD should always link to master?  The only odd one I know of is rOPUP (they use `production` as thei...
[20:12:55] <wikibugs>	 05Gitblit-Deprecate, 10Diffusion: Redirect git.wikimedia.org HEAD URLs to Diffusion - https://phabricator.wikimedia.org/T141965#2523784 (10Danny_B) Production is there default branch as master is in other repos, so it would work the same. I should have mention that I mean master or any default branch. (Backgro...
[20:20:01] <paladox>	 ostriches something like https://gerrit.wikimedia.org/r/#/c/302982/
[20:20:17] <paladox>	 Yes it needs a commit msg, but i will do it after you say yes or no
[20:21:08] <ostriches>	 I think it's confusing to follow, plus duplication.
[20:21:16] <ostriches>	 I'm thinking of splitting it into a couple of files.
[20:21:21] <ostriches>	 And use Include directives.
[20:21:27] <paladox>	 Ok
[20:21:54] <paladox>	 ostriches we could just create a different file with the http stuff only
[20:22:37] <paladox>	 Oh, your idea sounds better, are you working on it now or sometime later?
[20:23:15] <wikibugs>	 10Continuous-Integration-Infrastructure, 07Tracking: Have unit tests of all wmf deployed extensions pass when installed together, in both PHP-Zend and HHVM (tracking) - https://phabricator.wikimedia.org/T69216#2523828 (10MBinder_WMF)
[20:24:05] <wikibugs>	 10Continuous-Integration-Infrastructure, 07Tracking: Have unit tests of all wmf deployed extensions pass when installed together, in both PHP-Zend and HHVM (tracking) - https://phabricator.wikimedia.org/T69216#2523832 (10Legoktm)
[20:25:39] <wikibugs>	 10Browser-Tests-Infrastructure, 10MediaWiki-extensions-MultimediaViewer, 06Reading-Web-Backlog, 03Reading-Web-Sprint-78-Terminal-Velocity: A JSON text must at least contain two octets! (JSON::ParserError) in MultimediaViewer, Echo, Flow, RelatedArticles, Mo... - https://phabricator.wikimedia.org/T129483#2523839
[20:34:29] <wikibugs>	 05Gitblit-Deprecate, 10Diffusion: Redirect git.wikimedia.org HEAD URLs to Diffusion - https://phabricator.wikimedia.org/T141965#2523894 (10Danny_B) So if the above applies, then it's trivial.  I can't submit a patch now, so just here (somebody please patch it on behalf of me, thanks):  https://phabricator.wiki...
[20:37:35] <FlorianSW>	 ostriches: MediaWiki-announce is only for MediaWiki core updates, right? I think about lists where I should write to get as much as possible attention from third parties (because of https://phabricator.wikimedia.org/T142133) :]
[20:38:35] <ostriches>	 ConfirmEdit is bundled in core.
[20:38:38] <ostriches>	 I think that's a legit reason
[20:40:07] <FlorianSW>	 ostriches: ok, thanks, i'll use it :)
[20:40:30] <FlorianSW>	 I suppose anyone can post to this list and it's not moderated? :D
[20:40:49] <ostriches>	 No, everyone's moderated and it discards posts by default.
[20:40:56] <ostriches>	 I'll /msg you :)
[20:44:38] <wikibugs>	 10MediaWiki-Releasing, 10MediaWiki-Containers: Ready-to-use Docker package for MediaWiki - https://phabricator.wikimedia.org/T92826#1121273 (10TheGleep) I cannot see an "Issues" link on the github page, so I am posting here.  I am trying to install this image so that I can have the latest version (the synctree...
[20:44:46] <wikibugs>	 05Gitblit-Deprecate, 10Diffusion: Redirect git.wikimedia.org HEAD URLs to Diffusion - https://phabricator.wikimedia.org/T141965#2523995 (10Paladox) Submited here https://gerrit.wikimedia.org/r/#/c/302747/
[20:46:44] <wikibugs>	 06Release-Engineering-Team, 06Operations, 10Traffic, 07HTTPS: Retire gerrit.wikimedia.org SSL cert - https://phabricator.wikimedia.org/T142131#2524001 (10RobH) I used to habitually revoke old certificates, but I was advised against doing so indescriminately by @bblack.  My understanding is unless the priva...
[21:05:11] <wikibugs>	 10Browser-Tests-Infrastructure, 06Reading-Web-Backlog, 03Reading-Web-Sprint-78-Terminal-Velocity, 07Unplanned-Sprint-Work: Upgrade Selenium gem for various reading web opened extensions - https://phabricator.wikimedia.org/T142141#2524100 (10Jdlrobson)
[21:05:32] <icinga-wm>	 RECOVERY - puppet last run on scandium is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[21:05:42] <wikibugs>	 10Browser-Tests-Infrastructure, 06Reading-Web-Backlog, 03Reading-Web-Sprint-78-Terminal-Velocity, 07Unplanned-Sprint-Work: Upgrade Selenium gem for various reading web opened extensions - https://phabricator.wikimedia.org/T142141#2524100 (10Jdlrobson)
[21:10:05] <shinken-wm>	 PROBLEM - Free space - all mounts on deployment-eventlogging03 is CRITICAL: CRITICAL: deployment-prep.deployment-eventlogging03.diskspace._srv.byte_percentfree (<11.11%)
[21:50:18] <wmf-insecte>	 Yippee, build fixed!
[21:50:18] <wmf-insecte>	 Project selenium-MobileFrontend » chrome,beta,Linux,contintLabsSlave && UbuntuTrusty build #108: 09FIXED in 18 min: https://integration.wikimedia.org/ci/job/selenium-MobileFrontend/BROWSER=chrome,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=contintLabsSlave%20&&%20UbuntuTrusty/108/
[21:57:42] <wmf-insecte>	 Yippee, build fixed!
[21:57:43] <wmf-insecte>	 Project selenium-MobileFrontend » firefox,beta,Linux,contintLabsSlave && UbuntuTrusty build #108: 09FIXED in 26 min: https://integration.wikimedia.org/ci/job/selenium-MobileFrontend/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=contintLabsSlave%20&&%20UbuntuTrusty/108/
[22:11:54] <jdlrobson>	 thcipriani: we're greeeeen again :) https://integration.wikimedia.org/ci/view/Mobile/job/selenium-MobileFrontend/108/
[22:12:05] <thcipriani>	 jdlrobson: \o/
[22:12:19] <wikibugs>	 10Browser-Tests-Infrastructure, 06Reading-Web-Backlog, 03Reading-Web-Sprint-78-Terminal-Velocity, 07Unplanned-Sprint-Work: Upgrade Selenium gem for various reading web opened extensions - https://phabricator.wikimedia.org/T142141#2524330 (10Jdlrobson) a:05dduvall>03Jdlrobson MobileFrontend is green aga...
[22:12:26] <wikibugs>	 10Browser-Tests-Infrastructure, 06Reading-Web-Backlog, 03Reading-Web-Sprint-78-Terminal-Velocity, 07Unplanned-Sprint-Work: Upgrade Selenium gem for various reading web opened extensions - https://phabricator.wikimedia.org/T142141#2524332 (10Jdlrobson)
[22:18:39] <wikibugs>	 10Browser-Tests-Infrastructure, 10MediaWiki-extensions-MultimediaViewer, 06Reading-Web-Backlog, 13Patch-For-Review, and 2 others: A JSON text must at least contain two octets! (JSON::ParserError) in MultimediaViewer, Echo, Flow, RelatedArticles, MobileFront... - https://phabricator.wikimedia.org/T129483#2524381
[22:25:11] <wikibugs>	 10Browser-Tests-Infrastructure, 06Reading-Web-Backlog, 13Patch-For-Review, 03Reading-Web-Sprint-78-Terminal-Velocity, 07Unplanned-Sprint-Work: Upgrade Selenium gem for various reading web opened extensions - https://phabricator.wikimedia.org/T142141#2524393 (10dduvall) Remember to run `bundle install` be...
[22:29:56] <wikibugs>	 10Browser-Tests-Infrastructure, 10MediaWiki-extensions-MultimediaViewer, 06Reading-Web-Backlog, 13Patch-For-Review, and 2 others: A JSON text must at least contain two octets! (JSON::ParserError) in MultimediaViewer, Echo, Flow, RelatedArticles, MobileFront... - https://phabricator.wikimedia.org/T129483#2524403
[22:48:35] <wikibugs>	 10Beta-Cluster-Infrastructure, 10ChangeProp, 10Citoid, 10ContentTranslation-CXserver, 10VisualEditor: Merge service-specific instances in deployment-prep to the deployment-sca* instances - https://phabricator.wikimedia.org/T142100#2524472 (10greg) 05Open>03Invalid I'll rework :)
[22:49:36] <wikibugs>	 10Browser-Tests-Infrastructure, 06Reading-Web-Backlog, 13Patch-For-Review, 03Reading-Web-Sprint-78-Terminal-Velocity, 07Unplanned-Sprint-Work: Upgrade Selenium gem for various reading web opened extensions - https://phabricator.wikimedia.org/T142141#2524478 (10Jdlrobson) You're right. Thanks @dduvall for...
[22:49:43] <wikibugs>	 10Beta-Cluster-Infrastructure, 10Citoid, 10VisualEditor: Move citoid to deployment-sca* hosts in Beta Cluster - https://phabricator.wikimedia.org/T142150#2524479 (10greg)
[22:50:08] <wikibugs>	 10Beta-Cluster-Infrastructure, 10Citoid, 10VisualEditor: Move citoid to deployment-sca* hosts in Beta Cluster - https://phabricator.wikimedia.org/T142150#2524492 (10greg)
[22:51:48] <wikibugs>	 10Beta-Cluster-Infrastructure, 10Citoid, 10VisualEditor: Move citoid to deployment-sca* hosts in Beta Cluster - https://phabricator.wikimedia.org/T142150#2524479 (10greg)
[22:52:52] <wikibugs>	 10Beta-Cluster-Infrastructure, 10ContentTranslation-CXserver: Move apertium to deployment-sca* hosts in Beta Cluster - https://phabricator.wikimedia.org/T142152#2524511 (10greg)
[22:53:20] <wikibugs>	 10Beta-Cluster-Infrastructure, 10ContentTranslation-CXserver: Move apertium to deployment-sca* hosts in Beta Cluster - https://phabricator.wikimedia.org/T142152#2524523 (10greg)
[22:53:31] <wikibugs>	 10Beta-Cluster-Infrastructure, 10ContentTranslation-CXserver: Move apertium to deployment-sca* hosts in Beta Cluster - https://phabricator.wikimedia.org/T142152#2524511 (10greg)
[22:55:41] <wikibugs>	 10Beta-Cluster-Infrastructure, 10ChangeProp, 10Citoid, 10ContentTranslation-CXserver, 10VisualEditor: Merge service-specific instances in deployment-prep to the deployment-sca* instances - https://phabricator.wikimedia.org/T142100#2524532 (10greg) * Citoid: Created T142150, blocked by T107302 * Apertium:...
[23:02:41] <wikibugs>	 06Release-Engineering-Team, 06Operations, 15User-greg, 07Wikimedia-Incident: Institute quarterly(?) review of incident reports and follow-up - https://phabricator.wikimedia.org/T141287#2524577 (10ksmith) I don't really have the context to know what is going to make sense, so I'll toss out some ideas. Assum...
[23:18:18] <grrrit-wm>	 (03PS5) 10Awight: Use composer in DonationInterface hhvm tests [integration/config] - 10https://gerrit.wikimedia.org/r/301025 (https://phabricator.wikimedia.org/T141309) 
[23:24:31] <wikibugs>	 06Release-Engineering-Team, 15User-greg: Rework the #Beta-Cluster-Infrastructure workboard - https://phabricator.wikimedia.org/T138887#2524632 (10greg)
[23:31:40] <wikibugs>	 10Deployment-Systems, 06Operations: error on tin:/srv/mediawiki-staging: insufficient permission for adding an object to repository database .git/objects - https://phabricator.wikimedia.org/T127093#2524662 (10greg) 05Invalid>03Open This has been happening with more regularity lately :/  @Dzahn mentioned th...
[23:44:59] <wikibugs>	 06Release-Engineering-Team, 10Phabricator: Requesting access for @Danny_B to Phame - https://phabricator.wikimedia.org/T142142#2525291 (10Peachey88) > or at least its "Doing the needful" part  Why do you want access to the #release-engineering-team's [[ https://phabricator.wikimedia.org/phame/blog/view/1/ | bl...