[02:38:44] (03CR) 10Legoktm: [C: 032] Rename OpeningKeywordBracketSniff to OpeningKeywordParenthesisSniff [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/371759 (https://phabricator.wikimedia.org/T173273) (owner: 10Reedy) [02:39:26] (03Merged) 10jenkins-bot: Rename OpeningKeywordBracketSniff to OpeningKeywordParenthesisSniff [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/371759 (https://phabricator.wikimedia.org/T173273) (owner: 10Reedy) [02:43:45] Project beta-scap-eqiad build #168494: 04FAILURE in 0.44 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/168494/ [03:11:16] Yippee, build fixed! [03:11:17] Project beta-scap-eqiad build #168495: 09FIXED in 17 min: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/168495/ [04:47:50] PROBLEM - Mediawiki Error Rate on graphite-labs is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [10.0] [05:37:49] PROBLEM - Mediawiki Error Rate on graphite-labs is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [10.0] [06:10:04] PROBLEM - Free space - all mounts on deployment-fluorine02 is CRITICAL: CRITICAL: deployment-prep.deployment-fluorine02.diskspace._srv.byte_percentfree (<50.00%) [06:55:03] RECOVERY - Free space - all mounts on deployment-fluorine02 is OK: OK: All targets OK [07:13:40] PROBLEM - Puppet errors on deployment-sentry01 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [08:28:43] PROBLEM - Puppet errors on deployment-kafka03 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [08:52:51] RECOVERY - Mediawiki Error Rate on graphite-labs is OK: OK: Less than 1.00% above the threshold [1.0] [09:08:46] RECOVERY - Puppet errors on deployment-kafka03 is OK: OK: Less than 1.00% above the threshold [0.0] [09:46:36] !log maurelio@deployment-tin:/srv/mediawiki/dblists$ expanddblist flow-computed > /home/maurelio/flow-test.dblist (to test expandblist for a patch I am working on) [09:46:40] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [09:54:31] PROBLEM - Puppet errors on deployment-imagescaler02 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [10:04:36] PROBLEM - Puppet errors on integration-r-lang-01 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [10:25:33] can I temporary create on deployment-tin a securepollglobal-computed.dblist to run expanddblist afterwards? [10:25:41] I'm trying: [10:26:02] maurelio@deployment-tin:~$ expanddblist /home/maurelio/securepollglobal-computed > /home/maurelio/securepollglobal.dblist [10:26:32] and get: [10:26:35] maurelio@deployment-tin:~$ cat securepollglobal.dblist [10:26:36] [Mon Aug 14 10:24:46 2017] [hphp] [13636:7fe32bb06200:0:000001] [] [10:26:38] Fatal error: Uncaught exception 'Exception' with message 'MWWikiversions::readDbListFile(): unable to read /home/maurelio/securepollglobal-computed. [10:26:39] ' in /srv/mediawiki/multiversion/MWWikiversions.php:79 [10:26:41] Stack trace: [10:26:42] #0 /srv/mediawiki/multiversion/MWWikiversions.php(56): MWWikiversions::readDbListFile() [10:26:44] #1 /usr/local/bin/expanddblist(5): MWWikiversions::evalDbListExpression() [10:26:45] #2 {main} [10:34:32] RECOVERY - Puppet errors on deployment-imagescaler02 is OK: OK: Less than 1.00% above the threshold [0.0] [10:35:16] 10Continuous-Integration-Config, 10MinervaNeue, 10Readers-Web-Backlog: MinervaNeue mwext-doxygen-publish fails - https://phabricator.wikimedia.org/T173255#3523596 (10Jhernandez) [10:50:03] maurelio@deployment-tin:~$ expanddblist "%% all.dblist - private.dblist - fishbowl.dblist - closed.dblist - deleted.dblist - silver.dblist - wikimania.dblist" > /home/maurelio/spc4.dblist made it [10:50:12] but jenkins keeps complaining [10:50:16] :S [10:57:47] PROBLEM - Free space - all mounts on deployment-jobrunner02 is CRITICAL: CRITICAL: deployment-prep.deployment-jobrunner02.diskspace.root.byte_percentfree (<100.00%) [11:09:35] RECOVERY - Puppet errors on integration-r-lang-01 is OK: OK: Less than 1.00% above the threshold [0.0] [11:46:13] (03CR) 10Zfilipin: "Patch set #4 fixes whitespace in client.rb, but not in mediawiki_api.gemspec." [ruby/api] - 10https://gerrit.wikimedia.org/r/368609 (owner: 10Bekicot) [11:46:25] (03CR) 10Zfilipin: "recheck" [ruby/api] - 10https://gerrit.wikimedia.org/r/368609 (owner: 10Bekicot) [12:22:52] Yippee, build fixed! [12:22:52] Project selenium-GettingStarted » firefox,beta,Linux,BrowserTests build #494: 09FIXED in 51 sec: https://integration.wikimedia.org/ci/job/selenium-GettingStarted/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=BrowserTests/494/ [12:35:35] PROBLEM - Puppet errors on integration-r-lang-01 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [13:10:34] RECOVERY - Puppet errors on integration-r-lang-01 is OK: OK: Less than 1.00% above the threshold [0.0] [13:27:52] 10Release-Engineering-Team (Kanban), 10RelatedArticles, 10MW-1.30-release-notes (WMF-deploy-2017-08-08_(1.30.0-wmf.13)), 10Patch-For-Review, and 2 others: Rewrite Related pages browser tests in Node.js - https://phabricator.wikimedia.org/T164024#3523835 (10zeljkofilipin) a:03zeljkofilipin [13:43:29] (03CR) 10Zfilipin: [C: 032] WebdriverIO tests should look for LocalSettings.php in selenium folder [integration/config] - 10https://gerrit.wikimedia.org/r/370662 (https://phabricator.wikimedia.org/T164024) (owner: 10Jdlrobson) [13:44:34] (03Merged) 10jenkins-bot: WebdriverIO tests should look for LocalSettings.php in selenium folder [integration/config] - 10https://gerrit.wikimedia.org/r/370662 (https://phabricator.wikimedia.org/T164024) (owner: 10Jdlrobson) [13:45:01] Project selenium-VisualEditor » firefox,beta,Linux,BrowserTests build #491: 04FAILURE in 1 min 0 sec: https://integration.wikimedia.org/ci/job/selenium-VisualEditor/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=BrowserTests/491/ [13:56:51] (03CR) 10Zfilipin: "mediawiki-core-qunit-selenium-jessie job deployed:" [integration/config] - 10https://gerrit.wikimedia.org/r/370662 (https://phabricator.wikimedia.org/T164024) (owner: 10Jdlrobson) [15:01:36] PROBLEM - Puppet errors on integration-r-lang-01 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [15:33:15] 10MediaWiki-Codesniffer, 10Patch-For-Review: MediaWiki.WhiteSpace.OpeningKeywordBrace.WrongWhitespaceBeforeParenthesis unclear messages - https://phabricator.wikimedia.org/T173273#3524178 (10Umherirrender) 05Open>03Resolved p:05Triage>03Normal [15:38:13] 10Browser-Tests-Infrastructure, 10Release-Engineering-Team (Next), 10MinervaNeue, 10Readers-Web-Backlog, and 5 others: [4 hrs] MinervaNeue browser test are flaking (waiting for {:class=>"mw-notification", :tag_name=>"div"} to become present ) - https://phabricator.wikimedia.org/T170890#3524197 (10pmiazga)... [15:39:43] 10Release-Engineering-Team (Kanban), 10Phabricator: Custom task form for #mediawiki-extension-requests - https://phabricator.wikimedia.org/T160374#3096644 (10Kghbln) >>! In T160374#3521541, @MarcoAurelio wrote: > - Likewise, please switch the visibility of this form to members of #cleanup and remove it from th... [15:41:35] RECOVERY - Puppet errors on integration-r-lang-01 is OK: OK: Less than 1.00% above the threshold [0.0] [15:43:18] 10Release-Engineering-Team (Kanban), 10RelatedArticles, 10MW-1.30-release-notes (WMF-deploy-2017-08-08_(1.30.0-wmf.13)), 10Patch-For-Review, and 2 others: Rewrite Related pages browser tests in Node.js - https://phabricator.wikimedia.org/T164024#3524210 (10zeljkofilipin) [15:43:47] 10Release-Engineering-Team (Kanban), 10RelatedArticles, 10Readers-Web-Backlog (Tracking), 10User-zeljkofilipin: Create Jenkins job that runs RelatedArticles Selenium tests daily - https://phabricator.wikimedia.org/T171847#3524212 (10zeljkofilipin) a:03zeljkofilipin [16:02:36] PROBLEM - Puppet errors on integration-r-lang-01 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [16:08:06] 10Gerrit, 10Repository-Ownership-Requests: Review membership of "mediawiki" Gerrit group - https://phabricator.wikimedia.org/T168216#3358777 (10Florian) >>! In T168216#3369410, @MZMcBride wrote: > Who manages this group? The "Administrators" group: https://gerrit.wikimedia.org/r/#/admin/groups/1,members [16:11:38] (03CR) 10Florianschmidtwelzow: [C: 031] Archive Extension:ImageTagging [integration/config] - 10https://gerrit.wikimedia.org/r/371653 (https://phabricator.wikimedia.org/T167897) (owner: 10MarcoAurelio) [16:12:40] 10Release-Engineering-Team (Kanban), 10RelatedArticles, 10Readers-Web-Backlog (Tracking), 10User-zeljkofilipin: Create Jenkins job that runs RelatedArticles Selenium tests daily - https://phabricator.wikimedia.org/T171847#3524286 (10zeljkofilipin) >>! In T171847#3485253, @hashar wrote: > Add a parameter to... [16:17:18] 10Beta-Cluster-Infrastructure, 10MediaWiki-Authentication-and-authorization, 10MediaWiki-extensions-CentralAuth, 10MW-1.30-release-notes (WMF-deploy-2017-08-08_(1.30.0-wmf.13)), 10Patch-For-Review: "Loss of session data" on Beta Cluster - https://phabricator.wikimedia.org/T172560#3524331 (10Anomie) >>! I... [16:59:06] 10Release-Engineering-Team (Watching / External), 10JobRunner-Service, 10MediaWiki-Platform-Team, 10Operations, and 2 others: Collect error logs from jobchron/jobrunner services in Logstash - https://phabricator.wikimedia.org/T172479#3499719 (10Anomie) >>! In T172479#3502395, @greg wrote: > Adding #mediawi... [17:12:35] RECOVERY - Puppet errors on integration-r-lang-01 is OK: OK: Less than 1.00% above the threshold [0.0] [17:24:10] 10Release-Engineering-Team (Kanban), 10RelatedArticles, 10Readers-Web-Backlog (Tracking), 10User-zeljkofilipin: Create Jenkins job that runs RelatedArticles Selenium tests daily - https://phabricator.wikimedia.org/T171847#3524492 (10zeljkofilipin) Created another testing job: https://integration.wikimedia.... [17:44:02] 10Gerrit: Can not change group membership in gerrit as a group member anymore - https://phabricator.wikimedia.org/T173337#3524539 (10Florian) [17:44:49] 10Gerrit: Can not change group membership in gerrit as a group member anymore - https://phabricator.wikimedia.org/T173337#3524539 (10Zppix) You must ask repo admins to change the group ownership in gerrit to the project members. [17:45:00] 10Gerrit, 10Repository-Admins: Can not change group membership in gerrit as a group member anymore - https://phabricator.wikimedia.org/T173337#3524557 (10Zppix) [17:47:56] 10Beta-Cluster-Infrastructure, 10MediaWiki-Authentication-and-authorization, 10MediaWiki-extensions-CentralAuth, 10MW-1.30-release-notes (WMF-deploy-2017-08-08_(1.30.0-wmf.13)), 10Patch-For-Review: "Loss of session data" on Beta Cluster - https://phabricator.wikimedia.org/T172560#3524565 (10Etonkovidova)... [17:48:01] 10Gerrit, 10Repository-Admins: Can not change group membership in gerrit as a group member anymore - https://phabricator.wikimedia.org/T173337#3524539 (10Paladox) We removed the single group plugin. In gerrit the group has to own it's self for all members to be able to make changes to that group. [17:49:44] 10Gerrit, 10Repository-Admins: Can not change group membership in gerrit as a group member anymore - https://phabricator.wikimedia.org/T173337#3524572 (10Paladox) All new groups created for repos in gerrit will now own them selfs. All existing groups need to request this so an admin can change this. [18:02:05] 10Beta-Cluster-Infrastructure, 10MediaWiki-Authentication-and-authorization, 10MediaWiki-extensions-CentralAuth, 10MW-1.30-release-notes (WMF-deploy-2017-08-08_(1.30.0-wmf.13)), 10Patch-For-Review: "Loss of session data" on Beta Cluster - https://phabricator.wikimedia.org/T172560#3524587 (10Anomie) >>! I... [18:20:31] 10Release-Engineering-Team (Watching / External), 10Operations, 10Ops-Access-Requests, 10User-Addshore: Make @daniel a MediaWiki deployer - https://phabricator.wikimedia.org/T173230#3524636 (10greg) [18:20:57] 10Release-Engineering-Team (Next), 10Release Pipeline: Using helm to manage staging k8s applications - https://phabricator.wikimedia.org/T173129#3524638 (10greg) [18:21:01] 10Release-Engineering-Team (Next), 10Release Pipeline: Find CI container build location - https://phabricator.wikimedia.org/T173128#3524640 (10greg) [18:21:04] 10Release-Engineering-Team (Next), 10Release Pipeline (Blubber): Build mathoid container via blubber - https://phabricator.wikimedia.org/T173127#3524642 (10greg) [18:21:18] 10Release-Engineering-Team (Watching / External), 10Wikidata: Document wikidata deployment properly - https://phabricator.wikimedia.org/T168491#3524644 (10greg) [18:21:34] 10Release-Engineering-Team (Watching / External), 10JobRunner-Service, 10MediaWiki-Platform-Team, 10Operations, and 2 others: Collect error logs from jobchron/jobrunner services in Logstash - https://phabricator.wikimedia.org/T172479#3524646 (10aaron) Yeah that list should be updated. I happen to investiga... [18:30:54] 10Release-Engineering-Team (Watching / External), 10JobRunner-Service, 10Operations, 10Performance-Team, and 2 others: Collect error logs from jobchron/jobrunner services in Logstash - https://phabricator.wikimedia.org/T172479#3524653 (10greg) >>! In T172479#3524447, @Anomie wrote: >>>! In T172479#3502395,... [18:33:34] PROBLEM - Puppet errors on integration-r-lang-01 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [19:09:53] Yippee, build fixed! [19:09:54] Project selenium-MinervaNeue » chrome,beta,Linux,BrowserTests build #75: 09FIXED in 20 min: https://integration.wikimedia.org/ci/job/selenium-MinervaNeue/BROWSER=chrome,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=BrowserTests/75/ [19:13:35] RECOVERY - Puppet errors on integration-r-lang-01 is OK: OK: Less than 1.00% above the threshold [0.0] [19:19:40] Yippee, build fixed! [19:19:41] Project selenium-MinervaNeue » firefox,beta,Linux,BrowserTests build #75: 09FIXED in 30 min: https://integration.wikimedia.org/ci/job/selenium-MinervaNeue/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=BrowserTests/75/ [20:06:51] 10Release-Engineering-Team (Watching / External), 10JobRunner-Service, 10Performance-Team, 10Patch-For-Review, 10Regression: Investigate 30x increase in Jobrunner errors - https://phabricator.wikimedia.org/T171371#3524987 (10aaron) >>! In T171371#3521365, @Stashbot wrote: > {nav icon=file, name=Mentioned... [20:10:11] 10Release-Engineering-Team (Kanban), 10Phabricator: Custom task form for #mediawiki-extension-requests - https://phabricator.wikimedia.org/T160374#3524997 (10mmodell) Done. [20:10:21] 10Release-Engineering-Team (Kanban), 10Phabricator: Custom task form for #mediawiki-extension-requests - https://phabricator.wikimedia.org/T160374#3524998 (10mmodell) p:05Triage>03Normal [20:10:29] 10Release-Engineering-Team (Kanban), 10Phabricator: Custom task form for #mediawiki-extension-requests - https://phabricator.wikimedia.org/T160374#3096644 (10mmodell) 05Open>03Resolved [20:24:17] 10Release-Engineering-Team (Watching / External), 10JobRunner-Service, 10Performance-Team, 10Patch-For-Review, 10Regression: Investigate 30x increase in Jobrunner errors - https://phabricator.wikimedia.org/T171371#3525027 (10aaron) Error rate went from 500-1000/s to 50-80/s. [20:48:55] 10Release-Engineering-Team (Watching / External), 10JobRunner-Service, 10Performance-Team, 10Patch-For-Review, 10Regression: Investigate 30x increase in Jobrunner errors - https://phabricator.wikimedia.org/T171371#3525062 (10aaron) [20:49:28] 10Release-Engineering-Team (Watching / External), 10JobRunner-Service, 10Performance-Team, 10Patch-For-Review, 10Regression: Investigate 30x increase in Jobrunner errors - https://phabricator.wikimedia.org/T171371#3462630 (10aaron) [20:50:05] 10Release-Engineering-Team (Watching / External), 10JobRunner-Service, 10Performance-Team, 10Patch-For-Review, 10Regression: Investigate 30x increase in Jobrunner errors - https://phabricator.wikimedia.org/T171371#3462630 (10aaron) 05Open>03Resolved Closing. The two logging-related improvement action... [21:22:47] 10Release-Engineering-Team (Kanban): Identify Orphaned components/code - https://phabricator.wikimedia.org/T173349#3525095 (10Jrbranaa) [21:32:22] 10Release-Engineering-Team (Kanban), 10Technical-Debt: Setup Tech Debt SIG meetings - https://phabricator.wikimedia.org/T173351#3525133 (10Jrbranaa) [21:34:06] PROBLEM - Puppet errors on deployment-kafka01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [21:35:48] PROBLEM - Puppet errors on deployment-etcd-01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [21:36:48] PROBLEM - Puppet errors on deployment-pdf01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0]