[00:41:52] PROBLEM - Puppet errors on deployment-ores01 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [00:44:48] PROBLEM - Puppet errors on deployment-imagescaler02 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [01:02:29] PROBLEM - Puppet errors on deployment-cache-upload04 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [01:11:52] RECOVERY - Puppet errors on deployment-ores01 is OK: OK: Less than 1.00% above the threshold [0.0] [01:19:46] RECOVERY - Puppet errors on deployment-imagescaler02 is OK: OK: Less than 1.00% above the threshold [0.0] [01:37:31] RECOVERY - Puppet errors on deployment-cache-upload04 is OK: OK: Less than 1.00% above the threshold [0.0] [01:42:28] I have finally tracked down https://gerrit.wikimedia.org/r/#/c/435719/ [02:02:37] 10Phabricator: Option to Turn Off Status Updates in Phabricator Task-Threads - https://phabricator.wikimedia.org/T195728#4235323 (10Johnywhy) [02:15:29] 10Phabricator: Option to Turn Off Status Updates in Phabricator Task-Threads - https://phabricator.wikimedia.org/T195728#4235333 (10Johnywhy) [02:21:01] 10Phabricator: Option to Turn Off Status Updates in Phabricator Task-Threads - https://phabricator.wikimedia.org/T195728#4235334 (10Johnywhy) [04:47:53] !log running modified version of hashar's gear_client.py on contint1001, feel free to kill if it causes problems [04:47:56] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [05:38:04] 10Beta-Cluster-Infrastructure, 10Puppet, 10Tracking: Deployment-prep hosts with puppet errors (tracking) - https://phabricator.wikimedia.org/T132259#4235415 (10EddieGP) [05:38:10] 10Beta-Cluster-Infrastructure, 10Operations, 10media-storage, 10Patch-For-Review, 10Puppet: Puppet broken on deployment-ms-be0[34] with evaluation error in swift module - https://phabricator.wikimedia.org/T184236#4235414 (10EddieGP) 05Open>03Resolved [05:41:48] 10Beta-Cluster-Infrastructure, 10Services (next), 10User-Eevans: Reset RESTBase deployment-prep environment - https://phabricator.wikimedia.org/T186750#3954060 (10EddieGP) See also T186994, T195709, T192539. [06:41:37] 10Phabricator (Upstream), 10Upstream: Option to Turn Off Status Updates in Phabricator Task-Threads - https://phabricator.wikimedia.org/T195728#4235431 (10Aklapper) p:05Triage>03Lowest I neither expect upstream developers to implement this nor Wikimedia to maintain some custom patch so I'm afraid this won'... [07:00:15] PROBLEM - Puppet errors on deployment-restbase01 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [07:14:33] 10Release-Engineering-Team (Kanban), 10MediaWiki-extensions-MultimediaViewer, 10Multimedia, 10Quibble: quibble: with hhvm, could not load extension asset - https://phabricator.wikimedia.org/T195634#4235485 (10hashar) [07:35:14] RECOVERY - Puppet errors on deployment-restbase01 is OK: OK: Less than 1.00% above the threshold [0.0] [08:00:55] 10Release-Engineering-Team (Kanban), 10MediaWiki-extensions-MultimediaViewer, 10Multimedia, 10Quibble: quibble: with hhvm, could not load extension asset - https://phabricator.wikimedia.org/T195634#4235549 (10hashar) [08:18:00] 10Release-Engineering-Team (Kanban), 10MediaWiki-extensions-MultimediaViewer, 10Multimedia, 10Quibble: quibble: with hhvm, could not load extension asset - https://phabricator.wikimedia.org/T195634#4235605 (10hashar) Response headers with php7: < HTTP/1.1 200 OK < Host: 127.0.0.1:8888 < Connection: close... [08:29:38] 10Release-Engineering-Team (Kanban), 10MediaWiki-extensions-MultimediaViewer, 10Multimedia, 10Quibble, and 2 others: quibble: with hhvm, "could not load extension asset expand.svg" due to lack of proper Content-Type header - https://phabricator.wikimedia.org/T195634#4235613 (10hashar) [08:29:49] 10Release-Engineering-Team (Kanban), 10MediaWiki-extensions-MultimediaViewer, 10Multimedia, 10Quibble, and 2 others: quibble: with hhvm, "could not load extension asset expand.svg" due to lack of proper Content-Type header - https://phabricator.wikimedia.org/T195634#4233061 (10hashar) https://docs.hhvm.com... [08:30:01] 10Release-Engineering-Team (Kanban), 10MediaWiki-extensions-MultimediaViewer, 10Multimedia, 10Quibble, and 2 others: quibble: with hhvm, "could not load extension asset expand.svg" due to lack of proper Content-Type header - https://phabricator.wikimedia.org/T195634#4235616 (10hashar) a:03hashar [09:12:00] (03PS1) 10Hashar: hhvm server should set .svg Content-Type [integration/quibble] - 10https://gerrit.wikimedia.org/r/435748 (https://phabricator.wikimedia.org/T195634) [09:38:54] 10Project-Admins: Create a new phabricator project for Lingua Libre - https://phabricator.wikimedia.org/T195565#4235814 (10Aklapper) Ah, thanks! Looking at https://github.com/lingua-libre and seeing there are five subprojects makes me wonder * if you want to have one project in Wikimedia Phabricator (and e.g. [[... [09:41:32] 10Phabricator (Upstream), 10Upstream: Option to Turn Off Status Updates in Phabricator Task-Threads - https://phabricator.wikimedia.org/T195728#4235816 (10Johnywhy) >>! In T195728#4235431, @Aklapper wrote: > I neither expect upstream developers to implement this Because? If you decline this task, then don't... [09:43:49] 10Project-Admins: Create a new phabricator project for Lingua Libre - https://phabricator.wikimedia.org/T195565#4235824 (10Trizek) >>! In T195565#4235283, @0x010C wrote: > Huuuu (bis), I've completely forgotten this page... Lingua Libre is not ready (yet!) to welcome developers with little experience of MediaWik... [09:47:57] 10Phabricator, 10Project-Admins, 10Security-Team: Create Tag for WMDE Fundraising Security issues - https://phabricator.wikimedia.org/T194286#4235838 (10Aklapper) @gabriel-wmde: To hopefully clarify my somewhat dry comment in T194286#4194998, I think there are two options: * Use the "default" Security task w... [09:58:41] 10Project-Admins: Add #SVG tag to Phabricator - https://phabricator.wikimedia.org/T194188#4235849 (10Aklapper) @Volker_E: Could you provide a project description please? Please also see T194188#4192306 if you'd like to avoid triaging "The PNG preview of that SVG file I uploaded to Commons looks weird" tasks tha... [10:07:55] 10Phabricator (Upstream), 10Upstream: Option to Turn Off Status Updates in Phabricator Task-Threads - https://phabricator.wikimedia.org/T195728#4235893 (10Aklapper) @Johnywhy: Please check your tone here. No need to shoot the messenger. > Because? Because that is my understanding of upstream development's prio... [10:25:13] 10Phabricator (Upstream), 10Upstream: Option to Turn Off Status Updates in Phabricator Task-Threads - https://phabricator.wikimedia.org/T195728#4235907 (10Johnywhy) Your "human problem"-- excessive status messages-- is the byproduct of my **preferred** workflow. That preferred workflow is a //benefit// for me.... [10:35:21] (03PS5) 10Thiemo Kreuz (WMDE): Add FunctionAnnotations checking tags in function comments only [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/424554 (https://phabricator.wikimedia.org/T182057) [10:37:54] (03CR) 10Thiemo Kreuz (WMDE): "I think I fixed everything mentioned in the comments. Please have a look. Notably:" [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/424554 (https://phabricator.wikimedia.org/T182057) (owner: 10Thiemo Kreuz (WMDE)) [10:59:06] (03CR) 10Hashar: [C: 032] hhvm server should set .svg Content-Type [integration/quibble] - 10https://gerrit.wikimedia.org/r/435748 (https://phabricator.wikimedia.org/T195634) (owner: 10Hashar) [10:59:45] (03Merged) 10jenkins-bot: hhvm server should set .svg Content-Type [integration/quibble] - 10https://gerrit.wikimedia.org/r/435748 (https://phabricator.wikimedia.org/T195634) (owner: 10Hashar) [11:00:05] PROBLEM - Puppet errors on deployment-snapshot01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [11:00:16] (03CR) 10jenkins-bot: hhvm server should set .svg Content-Type [integration/quibble] - 10https://gerrit.wikimedia.org/r/435748 (https://phabricator.wikimedia.org/T195634) (owner: 10Hashar) [11:01:33] !log tagging quibble 0.0.13 hhvm server should set .svg Content-Type | T195634 [11:01:36] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [11:01:36] T195634: quibble: with hhvm, "could not load extension asset expand.svg" due to lack of proper Content-Type header - https://phabricator.wikimedia.org/T195634 [11:02:11] 10Project-Admins: Create a new phabricator project for Lingua Libre - https://phabricator.wikimedia.org/T195565#4236022 (10Sylvain_WMFr) >>! In T195565#4235264, @0x010C wrote: > Huuu, this is the repository of an old version of Lingua Libre, deprecated since one year. With the project grant linked in the descrip... [11:03:57] (03PS1) 10Hashar: docker: quibble 0.0.13 [integration/config] - 10https://gerrit.wikimedia.org/r/435767 (https://phabricator.wikimedia.org/T195634) [11:05:03] (03PS1) 10Hashar: Bump Quibble jobs to 0.0.13 [integration/config] - 10https://gerrit.wikimedia.org/r/435768 (https://phabricator.wikimedia.org/T195634) [11:07:15] (03CR) 10Hashar: [C: 032] docker: quibble 0.0.13 [integration/config] - 10https://gerrit.wikimedia.org/r/435767 (https://phabricator.wikimedia.org/T195634) (owner: 10Hashar) [11:08:46] (03Merged) 10jenkins-bot: docker: quibble 0.0.13 [integration/config] - 10https://gerrit.wikimedia.org/r/435767 (https://phabricator.wikimedia.org/T195634) (owner: 10Hashar) [11:09:25] !log Building Quibble Docker images 0.0.13 | T195634 [11:09:28] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [11:09:28] T195634: quibble: with hhvm, "could not load extension asset expand.svg" due to lack of proper Content-Type header - https://phabricator.wikimedia.org/T195634 [11:49:48] 10Project-Admins: Create a new phabricator project for Lingua Libre - https://phabricator.wikimedia.org/T195565#4236139 (100x010C) > * if you want to have one project in Wikimedia Phabricator (and e.g. [[ https://www.mediawiki.org/wiki/Phabricator/Project_management#Boards | use workboard columns ]] for categori... [11:57:59] (03CR) 10Hashar: [C: 032] "done" [integration/config] - 10https://gerrit.wikimedia.org/r/435768 (https://phabricator.wikimedia.org/T195634) (owner: 10Hashar) [11:59:34] (03Merged) 10jenkins-bot: Bump Quibble jobs to 0.0.13 [integration/config] - 10https://gerrit.wikimedia.org/r/435768 (https://phabricator.wikimedia.org/T195634) (owner: 10Hashar) [12:01:59] 10Project-Admins: Create a new phabricator project for Lingua Libre - https://phabricator.wikimedia.org/T195565#4236146 (10Aklapper) 05Open>03Resolved a:03Aklapper Sounds good! :) Requested public project #lingua_libre has been created: https://phabricator.wikimedia.org/project/view/3393/ Please encourag... [12:05:42] 10Beta-Cluster-Infrastructure, 10Services (later): Something is wrong with deployment-restbase02.deployment-prep.eqiad.wmflabs - https://phabricator.wikimedia.org/T195709#4236164 (10mobrovac) Not sure what that is about. Puppet runs fine, though. [12:08:03] 10Project-Admins: Create a new phabricator project for Lingua Libre - https://phabricator.wikimedia.org/T195565#4236171 (10Aklapper) @0x010C: Sorry, no [[ https://phabricator.wikimedia.org/project/manage/3393/ | green color ]] for that tag: https://www.mediawiki.org/wiki/Phabricator/Project_management#Types_of_P... [12:08:56] 10Release-Engineering-Team (Kanban), 10MediaWiki-extensions-MultimediaViewer, 10Multimedia, 10Quibble, and 3 others: quibble: with hhvm, "could not load extension asset expand.svg" due to lack of proper Content-Type header - https://phabricator.wikimedia.org/T195634#4236173 (10hashar) 05Open>03Resolved... [12:09:19] 10Project-Admins: Would like to create project Wikidocumentaries - https://phabricator.wikimedia.org/T195391#4236176 (10Aklapper) 05Open>03Resolved a:03Aklapper Requested public project #Wikidocumentaries has been created: https://phabricator.wikimedia.org/tag/wikidocumentaries/ Please encourage intereste... [12:09:46] 10Release-Engineering-Team (Kanban), 10MediaWiki-extensions-MultimediaViewer, 10Multimedia, 10Quibble, and 3 others: quibble: with hhvm, "could not load extension asset expand.svg" due to lack of proper Content-Type header - https://phabricator.wikimedia.org/T195634#4236180 (10hashar) [12:09:51] 10Continuous-Integration-Infrastructure (shipyard), 10Release-Engineering-Team (Kanban), 10releng-201718-q3, 10Epic, 10Patch-For-Review: [EPIC] Migrate Mediawiki jobs from Nodepool to Docker - https://phabricator.wikimedia.org/T183512#4236179 (10hashar) [12:10:26] 10Continuous-Integration-Infrastructure (shipyard), 10Release-Engineering-Team (Kanban), 10releng-201718-q3, 10Epic, 10Patch-For-Review: [EPIC] Migrate Mediawiki jobs from Nodepool to Docker - https://phabricator.wikimedia.org/T183512#4168008 (10hashar) [12:14:12] 10Project-Admins: Create a new phabricator project for Lingua Libre - https://phabricator.wikimedia.org/T195565#4230965 (100x010C) @Aklapper Thanks a lot for your comments and tour help! :) [12:43:19] 10Continuous-Integration-Infrastructure (shipyard), 10Release-Engineering-Team (Kanban), 10releng-201718-q3, 10Epic, 10Patch-For-Review: [EPIC] Migrate Mediawiki jobs from Nodepool to Docker - https://phabricator.wikimedia.org/T183512#4236299 (10hashar) [12:43:38] 10Continuous-Integration-Infrastructure (shipyard), 10Release-Engineering-Team (Kanban), 10releng-201718-q3, 10Epic, 10Patch-For-Review: [EPIC] Migrate Mediawiki jobs from Nodepool to Docker - https://phabricator.wikimedia.org/T183512#4168076 (10hashar) [12:50:03] 10Continuous-Integration-Infrastructure (shipyard), 10Release-Engineering-Team (Kanban), 10releng-201718-q3, 10Epic, 10Patch-For-Review: [EPIC] Migrate Mediawiki jobs from Nodepool to Docker - https://phabricator.wikimedia.org/T183512#4236309 (10hashar) [12:50:22] 10Continuous-Integration-Infrastructure (shipyard), 10Release-Engineering-Team (Kanban), 10releng-201718-q3, 10Epic, 10Patch-For-Review: [EPIC] Migrate Mediawiki jobs from Nodepool to Docker - https://phabricator.wikimedia.org/T183512#4168120 (10hashar) [12:50:44] 10Continuous-Integration-Infrastructure (shipyard), 10Release-Engineering-Team (Kanban), 10releng-201718-q3, 10Epic, 10Patch-For-Review: [EPIC] Migrate Mediawiki jobs from Nodepool to Docker - https://phabricator.wikimedia.org/T183512#4168127 (10hashar) [13:18:46] PROBLEM - Puppet errors on deployment-maps03 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [13:31:06] 10Phabricator: Phabricator emails should have links to tasks before description - https://phabricator.wikimedia.org/T195771#4236450 (10jhsoby) [13:31:45] 10Phabricator (Upstream), 10Upstream: Phabricator emails should have links to tasks before description - https://phabricator.wikimedia.org/T195771#4236465 (10jhsoby) Probably upstream when I think about it. [13:37:36] 10Phabricator (Upstream), 10Upstream: Phabricator emails should have links to tasks before description - https://phabricator.wikimedia.org/T195771#4236485 (10Aklapper) p:05Triage>03Lowest Also see https://secure.phabricator.com/T5244#86775 [13:40:44] 10Phabricator (Upstream), 10Upstream: Phabricator emails should have links to tasks before description - https://phabricator.wikimedia.org/T195771#4236493 (10Aklapper) I am tempted to decline this request (though I sometimes also suffer from this). If you set up notifications to get notified about changes, you... [13:49:40] 10Phabricator, 10Release-Engineering-Team (Kanban), 10Operations, 10Patch-For-Review, and 2 others: Apache on phab1001 is gradually leaking worker processes which are stuck in "Gracefully finishing" state - https://phabricator.wikimedia.org/T182832#4236538 (10elukey) The plan is to reimage phab1001 to Stre... [15:37:35] 10Phabricator, 10Developer-Relations (Apr-Jun-2018): Try to identify new developers (via assignee field) in Phab tasks and potentially follow up - https://phabricator.wikimedia.org/T195780#4236793 (10Aklapper) [16:04:48] 10Phabricator, 10Developer-Relations (Apr-Jun-2018): Try to identify new developers (via assignee field) in Phab tasks and potentially follow up - https://phabricator.wikimedia.org/T195780#4236856 (10Aklapper) ``` SELECT DISTINCT CONCAT("https://phabricator.wikimedia.org/p/", u.userName) AS userName FROM phabr... [16:09:38] PROBLEM - Long lived cherry-picks on puppetmaster on deployment-puppetmaster03 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [16:41:27] 10Continuous-Integration-Infrastructure (shipyard), 10Release-Engineering-Team (Kanban), 10releng-201718-q3, 10Epic, 10Patch-For-Review: [EPIC] Migrate Mediawiki jobs from Nodepool to Docker - https://phabricator.wikimedia.org/T183512#4236920 (10hashar) [17:43:33] (2018-05-27 14:29:39 Krenair | WHy has someone started naming hosts with a hyphen before the number?) - because joe wanted to have different puppet prefixes for the new strech instances (with hyphen) and the old jessie ones (without hyphen) ;) [17:50:26] hashar: is there any reason I can't `import zuul` in python on contint1001? [17:53:09] eddiegp, oh, well I just made some stretch instances without hyphens [17:53:09] oops [17:58:23] Krenair: I think this was all about mediawiki instances (deployment-mediawiki06 is still jessie, deployment-mediawiki-07 is stretch), so should be all good :) [18:10:49] oh ok [18:10:55] bit of a strange convention [18:11:03] it's non-obvious so people may not apply it properly [18:11:07] people may not recognise it properly [18:13:13] and you can still sudo cumin 'name:"deployment-mediawiki.*"' 'lsb_release -c' [18:41:53] (03CR) 10Umherirrender: [C: 031] Make seccheck voting for RevisionSlider and ElectronPdfService [integration/config] - 10https://gerrit.wikimedia.org/r/434799 (owner: 10Legoktm) [18:43:39] legoktm: yeah Zuul is installed using a virtualenv [18:43:58] use /usr/share/python/zuul/bin/python [18:44:27] with modules being under /usr/share/python/zuul/lib/python2.7/site-packages/ [18:46:07] legoktm: and for tomorrow I found a nice corner case which is that extension.json AutoloadNamespaces is not loaded when extensions db tables are created :] https://phabricator.wikimedia.org/T195783 [18:55:06] legoktm: dont feel obliged to look at it today since it is an us holiday [18:57:00] legoktm: FIX IT NOW [18:57:01] OR ELSE [19:36:16] hashar: I'm on summer vacation, literally every day is a holiday to me ^.^ [19:53:55] legoktm: sounds great :] [19:55:00] then tomorrow I will try to figure out "Creating administrator user account" which dies when trying to reach the centralauth table that hasn't been created yet :^p [19:56:32] legoktm hi, wondering if you could re review https://gerrit.wikimedia.org/r/#/c/435689/ please? :) [19:56:38] (addressed your feedback) [20:43:50] PROBLEM - Puppet errors on integration-slave-jessie-android is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [21:08:22] PROBLEM - Puppet errors on deployment-mx02 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [21:47:09] PROBLEM - SSH on integration-slave-docker-1014 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:56:58] RECOVERY - SSH on integration-slave-docker-1014 is OK: SSH OK - OpenSSH_6.7p1 Debian-5+deb8u4 (protocol 2.0) [22:57:12] ok, I figured out the installer/autoloader issue [22:57:25] I'm just unsure how no one has noticed it was broken until now