[01:42:30] 10Scap (Scap3-Adoption-Phase1), 10releng-201516-q2, 10releng-201516-q3, 10releng-201516-q4, and 2 others: [keyresult] Migrate all Service team owned services to scap - https://phabricator.wikimedia.org/T109926#3517829 (10greg) [01:42:40] 10Scap (Scap3-MediaWiki-MVP), 10releng-201516-q3, 10releng-201617-q4, 10scap2, and 2 others: [EPIC] Migrate the MW weekly train deploy to scap3 - https://phabricator.wikimedia.org/T114313#3517832 (10greg) [01:42:43] 10Scap (Scap3-Adoption-Phase1), 10releng-201516-q2, 10releng-201516-q3, 10releng-201516-q4, and 2 others: [keyresult] Migrate all Service team owned services to scap - https://phabricator.wikimedia.org/T109926#1562968 (10greg) [01:43:29] 10Scap (Scap3-Adoption-Phase1), 10releng-201516-q2, 10releng-201516-q3, 10releng-201516-q4, and 2 others: [keyresult] Migrate all Service team owned services to scap - https://phabricator.wikimedia.org/T109926#1562968 (10greg) 05Open>03Resolved a:03mobrovac Thanks @mobrovac \o/ [01:58:07] 10MediaWiki-Codesniffer: Always use leading zeros in php files for floats - https://phabricator.wikimedia.org/T170442#3517844 (10Legoktm) [02:02:24] (03PS1) 10Legoktm: Add sniff to forbid PHP 7 scalar type hints [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/371219 (https://phabricator.wikimedia.org/T171335) [02:18:55] (03CR) 10Reedy: [C: 032] Add sniff to forbid PHP 7 scalar type hints [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/371219 (https://phabricator.wikimedia.org/T171335) (owner: 10Legoktm) [02:19:00] (03Merged) 10jenkins-bot: Add sniff to forbid PHP 7 scalar type hints [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/371219 (https://phabricator.wikimedia.org/T171335) (owner: 10Legoktm) [02:21:25] (03CR) 10Legoktm: [C: 032] Release 0.11.0 [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/371220 (owner: 10Legoktm) [02:22:01] (03Merged) 10jenkins-bot: Release 0.11.0 [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/371220 (owner: 10Legoktm) [02:27:39] (03CR) 10Krinkle: ""00:00:01.538 Error: debian/changelog could not be read. Is this really a Debian package?" [integration/uprightdiff] - 10https://gerrit.wikimedia.org/r/371201 (owner: 10Legoktm) [02:28:40] Krinkle: https://gerrit.wikimedia.org/r/#/c/370695/ [02:29:22] 10MediaWiki-Codesniffer: Add Sniff to forbid PHP 7+ type hints - https://phabricator.wikimedia.org/T171335#3517855 (10Ricordisamoa) [03:18:27] PROBLEM - Work requests waiting in Zuul Gearman server https://grafana.wikimedia.org/dashboard/db/zuul-gearman on contint1001 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [140.0] [03:37:02] PROBLEM - Free space - all mounts on deployment-fluorine02 is CRITICAL: CRITICAL: deployment-prep.deployment-fluorine02.diskspace._srv.byte_percentfree (<20.00%) [04:04:19] 10Gerrit: mediawiki/extensions/MoveToCommons is read only but files are not archived - https://phabricator.wikimedia.org/T173076#3517885 (10Legoktm) [04:05:17] 10Gerrit: mediawiki/extensions/MoveToCommons is read only but files are not archived - https://phabricator.wikimedia.org/T173076#3517898 (10Legoktm) Same with MoveToCommonsClient :( [04:11:38] Project selenium-MultimediaViewer » safari,beta,OS X 10.9,BrowserTests build #482: 04FAILURE in 15 min: https://integration.wikimedia.org/ci/job/selenium-MultimediaViewer/BROWSER=safari,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=OS%20X%2010.9,label=BrowserTests/482/ [05:49:48] PROBLEM - Mediawiki Error Rate on graphite-labs is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [10.0] [06:26:07] 10Continuous-Integration-Infrastructure, 10Nodepool: Increase Jenkins/Nodepool capacity - https://phabricator.wikimedia.org/T173047#3517917 (10bd808) The primary issue from the point of view of WMCS is that as instance creation/deletion rates with a higher Nodepool quota it causes core OpenStack components (sp... [06:49:48] PROBLEM - Mediawiki Error Rate on graphite-labs is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [10.0] [06:57:00] RECOVERY - Free space - all mounts on deployment-fluorine02 is OK: OK: All targets OK [08:09:49] RECOVERY - Mediawiki Error Rate on graphite-labs is OK: OK: Less than 1.00% above the threshold [1.0] [08:39:31] 10Browser-Tests-Infrastructure, 10Release-Engineering-Team (Kanban), 10VisualEditor: add VE tests for all "Insert" items - https://phabricator.wikimedia.org/T65708#3517963 (10zeljkofilipin) p:05Unbreak!>03Lowest [08:46:49] 10Gerrit, 10Cleanup: mediawiki/extensions/MoveToCommons is read only but files are not archived - https://phabricator.wikimedia.org/T173076#3517976 (10Aklapper) [10:08:07] RECOVERY - Work requests waiting in Zuul Gearman server https://grafana.wikimedia.org/dashboard/db/zuul-gearman on contint1001 is OK: OK: Less than 30.00% above the threshold [90.0] [11:21:13] PROBLEM - Puppet errors on deployment-sca02 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [12:22:59] Yippee, build fixed! [12:23:00] Project selenium-GettingStarted » firefox,beta,Linux,BrowserTests build #491: 09FIXED in 58 sec: https://integration.wikimedia.org/ci/job/selenium-GettingStarted/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=BrowserTests/491/ [13:34:31] PROBLEM - Puppet errors on deployment-eventlogging04 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [13:46:43] Yippee, build fixed! [13:46:44] Project selenium-VisualEditor » firefox,beta,Linux,BrowserTests build #488: 09FIXED in 2 min 43 sec: https://integration.wikimedia.org/ci/job/selenium-VisualEditor/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=BrowserTests/488/ [13:52:29] 10Scap, 10Analytics-Kanban, 10EventBus, 10Patch-For-Review, 10User-Elukey: eventlogging-service-eventbus scap deployments should depool/pool during deployment - https://phabricator.wikimedia.org/T171506#3518410 (10elukey) Marko just deployed to all the nodes, all good! (task will be closed soon as part o... [14:00:20] 10Continuous-Integration-Infrastructure, 10MediaWiki-Codesniffer: Use phpcs.xml of core in mediawiki-core-phpcs-jessie - https://phabricator.wikimedia.org/T170159#3518431 (10Umherirrender) >>! In T170159#3479284, @Fomafix wrote: > Seams to be fixed. https://gerrit.wikimedia.org/r/#/c/251312/36 reports no error... [14:07:48] (03PS1) 10Umherirrender: Change bootstrap-ci.php to match PHP CodeSniffer 3.0.0 [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/371475 (https://phabricator.wikimedia.org/T172933) [14:08:47] (03CR) 10Umherirrender: PHP CodeSniffer on CI should only lint HEAD (031 comment) [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/358554 (https://phabricator.wikimedia.org/T158974) (owner: 10Hashar) [14:10:33] (03PS2) 10Umherirrender: Change bootstrap-ci.php to match PHP CodeSniffer 3.0.0 [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/371475 (https://phabricator.wikimedia.org/T172933) [14:14:25] (03CR) 10jerkins-bot: [V: 04-1] Change bootstrap-ci.php to match PHP CodeSniffer 3.0.0 [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/371475 (https://phabricator.wikimedia.org/T172933) (owner: 10Umherirrender) [14:15:03] 10MediaWiki-Codesniffer: Add tests for bootstrap-ci.php in mediawiki-codesniffer - https://phabricator.wikimedia.org/T173095#3518460 (10Umherirrender) [14:15:14] (03CR) 10Umherirrender: "Created T173095 to track unit tests" [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/371475 (https://phabricator.wikimedia.org/T172933) (owner: 10Umherirrender) [14:16:33] (03PS3) 10Umherirrender: Change bootstrap-ci.php to match PHP CodeSniffer 3.0.0 [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/371475 (https://phabricator.wikimedia.org/T172933) [14:58:25] twentyafterfour hi, there was a phab security release relating to the recent git updates [14:58:26] https://secure.phabricator.com/T12961 [15:02:53] twentyafterfour i filled https://phabricator.wikimedia.org/T173099 [15:05:24] paladox: yep, when he's online he'll get to it [15:05:31] thanks :) [15:05:45] paladox: don't worry, we knew and talked about it last night [15:06:29] yep, just filled a task in case no one did. /me just found out now heh. [15:06:45] there's not always a point to filing tasks, ftr [15:06:57] there's nothing to discuss or track other than "doing it" [15:07:05] paladox: also, be sure to use his work account for work things [15:07:09] ok [15:07:17] @mmodell vs @20after4 [15:07:20] oh [15:07:41] oh i forgot about that [15:32:45] PROBLEM - Puppet errors on deployment-kafka03 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [15:33:09] RECOVERY - Long lived cherry-picks on puppetmaster on deployment-puppetmaster02 is OK: OK: Less than 100.00% above the threshold [0.0] [15:33:59] 10MediaWiki-Releasing, 10MediaWiki-Containers, 10Wikimania-Hackathon-2017, 10Services (doing), and 2 others: Ready-to-use Docker package for MediaWiki - https://phabricator.wikimedia.org/T92826#3518629 (10dbarratt) Docker would prefer for the docker code to be in the `wikimedia` org on GitHub: https://gith... [16:07:46] RECOVERY - Puppet errors on deployment-kafka03 is OK: OK: Less than 1.00% above the threshold [0.0] [16:18:08] 10MediaWiki-Codesniffer, 10MW-1.30-release-notes, 10MediaWiki-General-or-Unknown, 10Patch-For-Review: MediaWiki core was not passing phpcs - https://phabricator.wikimedia.org/T172933#3518706 (10Legoktm) Aw crap :( thanks for figuring this out... [17:20:32] 10Release-Engineering-Team (Kanban), 10Patch-For-Review, 10Release, 10Train Deployments: 1.30.0-wmf.12 deployment blockers - https://phabricator.wikimedia.org/T168053#3518885 (10mmodell) 05Open>03Resolved [18:07:26] 10Release-Engineering-Team, 10Release Pipeline (Blubber): Build mathoid container via blubber - https://phabricator.wikimedia.org/T173127#3519160 (10thcipriani) [18:07:58] 10Release-Engineering-Team, 10Release Pipeline (Blubber): Build mathoid container via blubber - https://phabricator.wikimedia.org/T173127#3519173 (10thcipriani) [18:09:37] 10Continuous-Integration-Config, 10commit-message-validator, 10Pywikibot-core, 10Patch-For-Review: Check the style of the commit message - https://phabricator.wikimedia.org/T109119#1540699 (10Dalba) Apparently [[ https://gerrit.wikimedia.org/r/368793 | my previous patch ]] is not working as expected. For... [18:10:41] 10Release-Engineering-Team, 10Release Pipeline: Find CI container build location - https://phabricator.wikimedia.org/T173128#3519177 (10thcipriani) [18:11:07] 10Release-Engineering-Team, 10Release Pipeline: Find CI container build location - https://phabricator.wikimedia.org/T173128#3519189 (10thcipriani) [18:11:28] 10Release-Engineering-Team, 10Release Pipeline: Find CI container build location - https://phabricator.wikimedia.org/T173128#3519177 (10thcipriani) [18:13:06] 10Release-Engineering-Team, 10Release Pipeline (Blubber): Build mathoid container via blubber - https://phabricator.wikimedia.org/T173127#3519196 (10thcipriani) [18:18:05] 10Release-Engineering-Team (Kanban), 10releng-201718-q1, 10Mathoid: Define functional tests for Mathoid running on the staging Kubernetes cluster for use in future gating decisions - https://phabricator.wikimedia.org/T170482#3519205 (10thcipriani) [18:18:28] 10Release-Engineering-Team (Kanban), 10releng-201718-q1, 10Mathoid, 10Release Pipeline: Define functional tests for Mathoid running on the staging Kubernetes cluster for use in future gating decisions - https://phabricator.wikimedia.org/T170482#3432925 (10thcipriani) [18:18:54] 10Release-Engineering-Team (Kanban), 10releng-201718-q1: Define method for monitoring and reacting to the mathoid functional tests - https://phabricator.wikimedia.org/T170483#3519208 (10thcipriani) [18:19:07] 10Release-Engineering-Team (Kanban), 10releng-201718-q1, 10Release Pipeline: Define method for monitoring and reacting to the mathoid functional tests - https://phabricator.wikimedia.org/T170483#3432940 (10thcipriani) [18:25:00] 10Release-Engineering-Team, 10Release Pipeline: Using helm to manage staging k8s applications - https://phabricator.wikimedia.org/T173129#3519225 (10thcipriani) [18:36:40] PROBLEM - App Server Main HTTP Response on deployment-mediawiki04 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:40:42] PROBLEM - English Wikipedia Mobile Main page on beta-cluster is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:41:30] RECOVERY - App Server Main HTTP Response on deployment-mediawiki04 is OK: HTTP OK: HTTP/1.1 200 OK - 50328 bytes in 0.794 second response time [18:45:35] RECOVERY - English Wikipedia Mobile Main page on beta-cluster is OK: HTTP OK: HTTP/1.1 200 OK - 39216 bytes in 0.930 second response time [18:57:34] (03Restored) 10TheDJ: Add Timeless skin [tools/release] - 10https://gerrit.wikimedia.org/r/344789 (https://phabricator.wikimedia.org/T160643) (owner: 10Dereckson) [18:57:57] (03PS2) 10TheDJ: Add Timeless skin [tools/release] - 10https://gerrit.wikimedia.org/r/344789 (https://phabricator.wikimedia.org/T154371) (owner: 10Dereckson) [19:01:18] (03PS3) 10TheDJ: Add Timeless skin [tools/release] - 10https://gerrit.wikimedia.org/r/344789 (https://phabricator.wikimedia.org/T154371) (owner: 10Dereckson) [19:02:01] (03PS4) 10TheDJ: Add Timeless skin [tools/release] - 10https://gerrit.wikimedia.org/r/344789 (https://phabricator.wikimedia.org/T154371) (owner: 10Dereckson) [19:02:03] thedj: you're quick ;) [19:12:41] Project selenium-MinervaNeue-BugFix » chrome,beta,Linux,BrowserTests build #28: 04FAILURE in 22 min: https://integration.wikimedia.org/ci/job/selenium-MinervaNeue-BugFix/BROWSER=chrome,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=BrowserTests/28/ [19:13:14] Project selenium-MinervaNeue » chrome,beta,Linux,BrowserTests build #71: 04FAILURE in 24 min: https://integration.wikimedia.org/ci/job/selenium-MinervaNeue/BROWSER=chrome,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=BrowserTests/71/ [19:22:25] Project selenium-MinervaNeue » firefox,beta,Linux,BrowserTests build #71: 15ABORTED in 33 min: https://integration.wikimedia.org/ci/job/selenium-MinervaNeue/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=BrowserTests/71/ [19:33:06] PROBLEM - Free space - all mounts on integration-slave-jessie-android is CRITICAL: CRITICAL: integration.integration-slave-jessie-android.diskspace._mnt.byte_percentfree (No valid datapoints found)integration.integration-slave-jessie-android.diskspace.root.byte_percentfree (<11.11%) [19:56:48] (03PS1) 10Legoktm: make-wmf-branch: Add Timeless [tools/release] - 10https://gerrit.wikimedia.org/r/371549 [19:58:19] (03CR) 10Legoktm: [C: 032] make-wmf-branch: Add Timeless [tools/release] - 10https://gerrit.wikimedia.org/r/371549 (owner: 10Legoktm) [20:04:50] (03Merged) 10jenkins-bot: make-wmf-branch: Add Timeless [tools/release] - 10https://gerrit.wikimedia.org/r/371549 (owner: 10Legoktm) [20:24:28] PROBLEM - Puppet errors on deployment-tin is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [20:25:30] (03Abandoned) 10Legoktm: Add Timeless skin [tools/release] - 10https://gerrit.wikimedia.org/r/344789 (https://phabricator.wikimedia.org/T154371) (owner: 10Dereckson) [20:25:57] !log added mediawiki::maintenance::wikidata to deployment-tin [20:26:00] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [20:34:26] RECOVERY - Puppet errors on deployment-tin is OK: OK: Less than 1.00% above the threshold [0.0] [20:56:20] RECOVERY - Puppet errors on deployment-ms-fe02 is OK: OK: Less than 1.00% above the threshold [0.0] [21:10:02] 10Scap: scap didn't stop after error rate hit 89% - https://phabricator.wikimedia.org/T173146#3519703 (10Legoktm) [21:48:14] 10Release-Engineering-Team (Watching / External), 10Wikimania-Hackathon-2017-Organization: Wikimania needs hosting on a server for onsite conference guide - https://phabricator.wikimedia.org/T172217#3519796 (10RobH) [22:07:13] Project selenium-MinervaNeue » firefox,beta,Linux,BrowserTests build #72: 04FAILURE in 30 min: https://integration.wikimedia.org/ci/job/selenium-MinervaNeue/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=BrowserTests/72/ [22:09:45] 10Scap: scap didn't stop after error rate hit 89% - https://phabricator.wikimedia.org/T173146#3519823 (10thcipriani) p:05Triage>03Normal This is expected behavior as of scap 3.6.0: https://github.com/wikimedia/scap/blob/master/scap/main.py#L108-L109 The canary check now relies on 2 instances being above an e... [22:25:04] 10Release-Engineering-Team, 10Wikidata: Document wikidata deployment properly - https://phabricator.wikimedia.org/T168491#3519856 (10hoo) Once T95663 is done this will hopefully not be a problem anymore.