[00:00:24] 10Deployment-Systems, 10Release-Engineering-Team (Next), 10Operations, 10Performance-Team, 10HHVM: Translation cache exhaustion caused by changes to PHP code in file scope - https://phabricator.wikimedia.org/T103886#3417383 (10greg) [00:00:40] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team: hhvm error logs on beta showing no errors when cluster down - https://phabricator.wikimedia.org/T151171#3417385 (10greg) 05Open>03stalled [00:01:50] 10Release-Engineering-Team (Kanban), 10releng-201617-q4, 10MediaWiki-General-or-Unknown, 10MW-1.29-release, 10Release: Release MediaWiki 1.29 - https://phabricator.wikimedia.org/T153271#3417386 (10demon) >>! In T153271#3417370, @mwjames wrote: >> All blockers are now resolved. > > Not sure how you come... [00:03:16] 10Continuous-Integration-Config, 10Release-Engineering-Team, 10Wikipedia-Android-App-Backlog, 10Spike, 10Technical-Debt: Investigate how to improve Android CI performance and stability - https://phabricator.wikimedia.org/T158014#3417389 (10greg) Update on stability? Performance is tracked/blocked by T15... [00:05:58] 10Release-Engineering-Team (Kanban), 10releng-201617-q4, 10MediaWiki-General-or-Unknown, 10MW-1.29-release, 10Release: Release MediaWiki 1.29 - https://phabricator.wikimedia.org/T153271#3417392 (10mwjames) > ... rc.1 has been released like 10 minutes ago, but it'll make it into the final release. I'm m... [00:06:03] PROBLEM - Puppet errors on integration-slave-docker-1000 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [00:18:32] 10Release-Engineering-Team (Kanban), 10releng-201617-q4, 10MediaWiki-General-or-Unknown, 10MW-1.29-release, 10Release: Release MediaWiki 1.29 - https://phabricator.wikimedia.org/T153271#3417403 (10demon) There's no query that can be done. See my comment on T167946#3417390 -- the mapping is complex and ha... [00:40:36] 10Release-Engineering-Team (Kanban), 10releng-201617-q4, 10MediaWiki-General-or-Unknown, 10MW-1.29-release, 10Release: Release MediaWiki 1.29 - https://phabricator.wikimedia.org/T153271#2874741 (10Legoktm) Theoretically if you searched for tasks tagged #mw-1.29-release without a #mw-1.29-release-notes ta... [00:41:04] RECOVERY - Puppet errors on integration-slave-docker-1000 is OK: OK: Less than 1.00% above the threshold [0.0] [00:43:39] 10Release-Engineering-Team (Kanban), 10releng-201617-q4, 10MediaWiki-General-or-Unknown, 10MW-1.29-release, 10Release: Release MediaWiki 1.29 - https://phabricator.wikimedia.org/T153271#3417455 (10Legoktm) (I think such a search would still have false positives and negatives, but it might be a bit closer) [00:47:19] 10Release-Engineering-Team (Kanban), 10releng-201617-q4, 10MediaWiki-General-or-Unknown, 10MW-1.29-release, 10Release: Release MediaWiki 1.29 - https://phabricator.wikimedia.org/T153271#3417456 (10MacFan4000) https://phabricator.wikimedia.org/project/board/2400/query/all/ [00:50:24] 10Release-Engineering-Team (Kanban), 10Wikimedia-Site-requests, 10Patch-For-Review: Update to interwiki map - https://phabricator.wikimedia.org/T169979#3417457 (10Billinghurst) :-) Thanks for prompt deployment [01:32:03] PROBLEM - Puppet errors on integration-slave-docker-1000 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [02:02:30] 10Browser-Tests-Infrastructure, 10Release-Engineering-Team (Next), 10MediaWiki-Core-Tests, 10Patch-For-Review, 10User-zeljkofilipin: WebdriverIO should run Chrome headlessly - https://phabricator.wikimedia.org/T167507#3417517 (10Krinkle) [02:12:02] RECOVERY - Puppet errors on integration-slave-docker-1000 is OK: OK: Less than 1.00% above the threshold [0.0] [02:12:37] 10Browser-Tests-Infrastructure, 10Continuous-Integration-Infrastructure: Upgrade to Chromium 59 or newer on Debian Jessie in CI - https://phabricator.wikimedia.org/T170032#3417522 (10Krinkle) [02:12:48] 10Browser-Tests-Infrastructure, 10Continuous-Integration-Infrastructure: Upgrade to Chromium 59 or newer on Debian Jessie in CI - https://phabricator.wikimedia.org/T170032#3417537 (10Krinkle) [02:12:50] 10Browser-Tests-Infrastructure, 10Release-Engineering-Team (Next), 10MediaWiki-Core-Tests, 10Patch-For-Review, 10User-zeljkofilipin: WebdriverIO should run Chrome headlessly - https://phabricator.wikimedia.org/T167507#3335241 (10Krinkle) [02:14:04] (03CR) 10Krinkle: "Blocked on T170032 (I assume)" [selenium] - 10https://gerrit.wikimedia.org/r/358578 (https://phabricator.wikimedia.org/T167507) (owner: 10Zfilipin) [02:22:34] 10Deployment-Systems, 10ArchCom-RfC, 10I18n: RFC: Reevaluate LocalisationUpdate extension for WMF - https://phabricator.wikimedia.org/T158360#3034951 (10Krinkle) [02:27:02] 10Continuous-Integration-Infrastructure, 10translatewiki.net, 10Patch-For-Review: L10n-bot should not force-merge / override Jenkins (breaks the build) - https://phabricator.wikimedia.org/T91707#3417562 (10Krinkle) [04:16:47] Yippee, build fixed! [04:16:47] Project selenium-MultimediaViewer » firefox,beta,Linux,BrowserTests build #446: 09FIXED in 20 min: https://integration.wikimedia.org/ci/job/selenium-MultimediaViewer/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=BrowserTests/446/ [04:38:11] 10Release-Engineering-Team (Kanban), 10releng-201617-q4, 10MediaWiki-General-or-Unknown, 10MW-1.29-release, 10Release: Release MediaWiki 1.29 - https://phabricator.wikimedia.org/T153271#3417680 (10greg) >>! In T153271#3417456, @MacFan4000 wrote: > > https://phabricator.wikimedia.org/project/board/2400/q... [04:42:27] Phab repo viewer has improved a fair bit. This looks quite nice: [04:42:28] https://phabricator.wikimedia.org/source/operations-puppet/history/production/modules/snapshot/manifests/cron [05:47:08] PROBLEM - Free space - all mounts on deployment-fluorine02 is CRITICAL: CRITICAL: deployment-prep.deployment-fluorine02.diskspace._srv.byte_percentfree (<22.22%) [07:07:11] RECOVERY - Free space - all mounts on deployment-fluorine02 is OK: OK: All targets OK [07:48:33] PROBLEM - Puppet errors on deployment-pdfrender02 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [08:23:32] RECOVERY - Puppet errors on deployment-pdfrender02 is OK: OK: Less than 1.00% above the threshold [0.0] [08:42:04] Hello, I need an emergency deploy for T170033 / https://gerrit.wikimedia.org/r/#/c/364011/ . It is a throttle rule requested a few of hours before the event date. I also asked at -operation [08:42:05] T170033: Temporarily Lift IP cap for account creation for Editathon in NYC - Saturday July 8th - https://phabricator.wikimedia.org/T170033 [09:44:33] PROBLEM - Puppet errors on deployment-pdfrender02 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [10:24:32] RECOVERY - Puppet errors on deployment-pdfrender02 is OK: OK: Less than 1.00% above the threshold [0.0] [13:46:27] Project selenium-VisualEditor » firefox,beta,Linux,BrowserTests build #453: 04FAILURE in 2 min 26 sec: https://integration.wikimedia.org/ci/job/selenium-VisualEditor/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=BrowserTests/453/ [14:22:23] PROBLEM - Puppet errors on castor is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [15:02:22] RECOVERY - Puppet errors on castor is OK: OK: Less than 1.00% above the threshold [0.0] [15:33:05] PROBLEM - Puppet errors on integration-slave-docker-1000 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [15:35:52] 10Release-Engineering-Team (Kanban), 10releng-201617-q4, 10MediaWiki-General-or-Unknown, 10MW-1.29-release, 10Release: Release MediaWiki 1.29 - https://phabricator.wikimedia.org/T153271#3418050 (10MacFan4000) Looks like it was al backported. What’s the next step for this task? [16:07:03] 10Release-Engineering-Team (Kanban), 10releng-201617-q4, 10MediaWiki-General-or-Unknown, 10MW-1.29-release, 10Release: Release MediaWiki 1.29 - https://phabricator.wikimedia.org/T153271#3418056 (10Reception123) @MacFan4000 You can see in mediawiki-announce that a release candidate has been issued. but... [16:13:12] PROBLEM - Puppet errors on deployment-ms-be03 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [16:20:34] PROBLEM - Puppet errors on deployment-ms-be04 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [17:19:35] 10Beta-Cluster-Infrastructure, 10MediaWiki-extensions-ORES, 10ORES, 10Scoring-platform-team, 10User-Ladsgroup: ORES spamming Beta Cluster's logstash - https://phabricator.wikimedia.org/T170026#3418110 (10Ladsgroup) a:03Ladsgroup [18:15:38] Project beta-scap-eqiad build #163144: 04FAILURE in 1 min 58 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/163144/ [18:26:15] Yippee, build fixed! [18:26:15] Project beta-scap-eqiad build #163145: 09FIXED in 2 min 27 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/163145/ [18:39:40] 10Release-Engineering-Team (Kanban), 10releng-201617-q4, 10MediaWiki-General-or-Unknown, 10MW-1.29-release, 10Release: Release MediaWiki 1.29 - https://phabricator.wikimedia.org/T153271#2874741 (10Xeriphas1994) For 1.28, it went like this: - First release candidate 2016-11-02 - Second release candid... [18:47:20] 10Release-Engineering-Team (Kanban), 10releng-201617-q4, 10MediaWiki-General-or-Unknown, 10MW-1.29-release, 10Release: Release MediaWiki 1.29 - https://phabricator.wikimedia.org/T153271#3418220 (10Reception123) I think both cases were not usual circumstances, as both releases were delayed. Hopefully no m... [19:44:55] 10Release-Engineering-Team (Kanban), 10Wikimedia-Site-requests, 10Patch-For-Review: Update to interwiki map - https://phabricator.wikimedia.org/T169979#3418242 (10Dereckson) @Billinghurst If you still have an sysop access on meta., don't forget to update the date when it's deployed by the way. [22:02:16] (03CR) 10Umherirrender: PHP CodeSniffer on CI should only lint HEAD (031 comment) [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/358554 (https://phabricator.wikimedia.org/T158974) (owner: 10Hashar)