[00:03:13] 10Browser-Tests-Infrastructure, 10MinervaNeue, 10Reading-Web-Backlog, 10Patch-For-Review: MinervaNeue browser test are flaking (waiting for {:class=>"mw-notification", :tag_name=>"div"} to become present ) - https://phabricator.wikimedia.org/T170890#3476780 (10Jdlrobson) I have a theory that I'm exploring:... [00:20:32] hi RainbowSprinkles [00:40:24] 10Browser-Tests-Infrastructure, 10Continuous-Integration-Config, 10Release-Engineering-Team (Kanban), 10Patch-For-Review: Browser test jobs should use xUnit publisher instead of Junit - https://phabricator.wikimedia.org/T94684#3476848 (10Jdlrobson) How can I see this in action? [01:52:43] PROBLEM - Puppet errors on deployment-mira is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [02:30:43] (03CR) 10Krinkle: [C: 031] "(Untested, code and test LGTM)" [integration/config] - 10https://gerrit.wikimedia.org/r/367691 (https://phabricator.wikimedia.org/T171352) (owner: 10Florianschmidtwelzow) [02:32:44] RECOVERY - Puppet errors on deployment-mira is OK: OK: Less than 1.00% above the threshold [0.0] [02:41:40] PROBLEM - Puppet errors on deployment-mathoid is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [03:06:51] 10Continuous-Integration-Config, 10Wikipedia-Android-App-Backlog: Don't re-download the SDK for every (non-periodic) CI job - https://phabricator.wikimedia.org/T171811#3476973 (10Mholloway) [03:07:06] 10Continuous-Integration-Config, 10Wikipedia-Android-App-Backlog: Don't re-download the Android SDK for every (non-periodic) CI job - https://phabricator.wikimedia.org/T171811#3476943 (10Mholloway) [03:21:42] RECOVERY - Puppet errors on deployment-mathoid is OK: OK: Less than 1.00% above the threshold [0.0] [04:01:46] Project selenium-MultimediaViewer » safari,beta,OS X 10.9,BrowserTests build #465: 04FAILURE in 5 min 45 sec: https://integration.wikimedia.org/ci/job/selenium-MultimediaViewer/BROWSER=safari,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=OS%20X%2010.9,label=BrowserTests/465/ [04:42:42] PROBLEM - Puppet errors on deployment-mathoid is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [05:22:42] RECOVERY - Puppet errors on deployment-mathoid is OK: OK: Less than 1.00% above the threshold [0.0] [05:44:10] (03CR) 10Reception123: [C: 04-1] "Minor, but the indent is not correctly done." [integration/config] - 10https://gerrit.wikimedia.org/r/367041 (https://phabricator.wikimedia.org/T171385) (owner: 10MacFan4000) [06:33:26] (03CR) 10XZise: [C: 04-1] "This shouldn't be merged at its current stage. The class isn't a real iterator at the moment but it also cannot be used in parallel or mul" [integration/commit-message-validator] - 10https://gerrit.wikimedia.org/r/367907 (owner: 10XZise) [07:08:08] RECOVERY - Free space - all mounts on deployment-fluorine02 is OK: OK: All targets OK [07:14:51] 10Scap, 10MediaWiki-extensions-LocalisationUpdate: l10nupdate failing with "git pull of extensions failed" since July 19th - https://phabricator.wikimedia.org/T171711#3477126 (10Amire80) >>! In T171711#3475616, @greg wrote: >>>! In T171711#3475383, @Amire80 wrote: >> Noticing a failure of a command in the log,... [07:15:23] 10Scap, 10MediaWiki-extensions-LocalisationUpdate: l10nupdate failing with "git pull of extensions failed" since July 19th - https://phabricator.wikimedia.org/T171711#3477127 (10Amire80) Is this issue supposed to be fixed? The message from the OP is still not updated in production. [07:43:41] PROBLEM - Puppet errors on deployment-mathoid is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [07:52:22] 10Release-Engineering-Team (Watching / External), 10Scap, 10ORES, 10Operations, 10Scoring-platform-team-Backlog: ORES should use git-fat for wheel deployments - https://phabricator.wikimedia.org/T171619#3477154 (10fgiunchedi) cc'ing #operations too [07:53:39] 10Beta-Cluster-Infrastructure, 10MediaWiki-extensions-WikibaseRepository, 10Wikidata: rebuildTermSqlIndex.php causing beta-update-databases-eqiad to timeout and abort - https://phabricator.wikimedia.org/T168036#3477158 (10WMDE-leszek) Should be "resolved" IMO. @Reedy: can you confirm? [08:11:00] 10Release-Engineering-Team, 10Reading-Web-Backlog, 10Wikimedia-Site-requests, 10Design (RW-Design-Debt), and 2 others: Deploy common Wordmarks - https://phabricator.wikimedia.org/T171556#3477190 (10hashar) Per @Krinkle on the Gerrit patch: > https://github.com/wikimedia/operations-mediawiki-config/commit/... [08:23:41] RECOVERY - Puppet errors on deployment-mathoid is OK: OK: Less than 1.00% above the threshold [0.0] [08:24:30] (03PS5) 10Hashar: Move wikimedia-fundraising-civicrm to Nodepool [integration/config] - 10https://gerrit.wikimedia.org/r/319331 (https://phabricator.wikimedia.org/T119140) [08:25:52] (03CR) 10Hashar: [C: 032] Move wikimedia-fundraising-civicrm to Nodepool [integration/config] - 10https://gerrit.wikimedia.org/r/319331 (https://phabricator.wikimedia.org/T119140) (owner: 10Hashar) [08:27:07] (03Merged) 10jenkins-bot: Move wikimedia-fundraising-civicrm to Nodepool [integration/config] - 10https://gerrit.wikimedia.org/r/319331 (https://phabricator.wikimedia.org/T119140) (owner: 10Hashar) [08:29:52] 10Deployment-Systems, 10Scap, 10Patch-For-Review: Update Debian Package for Scap3 - https://phabricator.wikimedia.org/T127762#3477245 (10fgiunchedi) 05Open>03Resolved @thcipriani all done! [08:37:58] (03PS1) 10Hashar: Revert "Move wikimedia-fundraising-civicrm to Nodepool" [integration/config] - 10https://gerrit.wikimedia.org/r/368145 [08:39:57] (03PS2) 10Hashar: Revert "Move wikimedia-fundraising-civicrm to Nodepool" [integration/config] - 10https://gerrit.wikimedia.org/r/368145 (https://phabricator.wikimedia.org/T119140) [08:42:24] (03PS1) 10Hashar: wikimedia-fundraising-civicrm-jessie as experimental job [integration/config] - 10https://gerrit.wikimedia.org/r/368148 (https://phabricator.wikimedia.org/T119140) [08:43:52] (03CR) 10Hashar: [C: 032] Revert "Move wikimedia-fundraising-civicrm to Nodepool" [integration/config] - 10https://gerrit.wikimedia.org/r/368145 (https://phabricator.wikimedia.org/T119140) (owner: 10Hashar) [08:43:56] (03CR) 10Hashar: [C: 032] wikimedia-fundraising-civicrm-jessie as experimental job [integration/config] - 10https://gerrit.wikimedia.org/r/368148 (https://phabricator.wikimedia.org/T119140) (owner: 10Hashar) [08:45:58] (03Merged) 10jenkins-bot: Revert "Move wikimedia-fundraising-civicrm to Nodepool" [integration/config] - 10https://gerrit.wikimedia.org/r/368145 (https://phabricator.wikimedia.org/T119140) (owner: 10Hashar) [08:46:07] (03Merged) 10jenkins-bot: wikimedia-fundraising-civicrm-jessie as experimental job [integration/config] - 10https://gerrit.wikimedia.org/r/368148 (https://phabricator.wikimedia.org/T119140) (owner: 10Hashar) [08:47:38] 10Continuous-Integration-Config, 10Continuous-Integration-Scaling, 10Release-Engineering-Team, 10releng-201516-q3, and 2 others: [keyresult] Migrate as many misc CI jobs as possible to Nodepool - https://phabricator.wikimedia.org/T119140#3477282 (10hashar) I blindly switched the `wikimedia-fundraising-civi... [09:10:27] (03PS7) 10Hashar: rm DeepSea [integration/config] - 10https://gerrit.wikimedia.org/r/367041 (https://phabricator.wikimedia.org/T171385) (owner: 10MacFan4000) [09:11:48] (03CR) 10Reception123: [C: 031] rm DeepSea [integration/config] - 10https://gerrit.wikimedia.org/r/367041 (https://phabricator.wikimedia.org/T171385) (owner: 10MacFan4000) [09:12:49] hashar: mind doing me a favor and checking out https://gerrit.wikimedia.org/r/#/c/348577/? (this has been an error for an extension needed for a long time) [09:12:57] (03CR) 10Hashar: [C: 032] "I have fixed the indentation issue (well caught Reception123 )." [integration/config] - 10https://gerrit.wikimedia.org/r/367041 (https://phabricator.wikimedia.org/T171385) (owner: 10MacFan4000) [09:13:50] (03Merged) 10jenkins-bot: rm DeepSea [integration/config] - 10https://gerrit.wikimedia.org/r/367041 (https://phabricator.wikimedia.org/T171385) (owner: 10MacFan4000) [09:39:15] (03CR) 10Hashar: [C: 032] "On the test/gerrit-ping repo change https://gerrit.wikimedia.org/r/#/c/226272/ I have filled a couple inline comments and commented reche" (032 comments) [integration/config] - 10https://gerrit.wikimedia.org/r/367691 (https://phabricator.wikimedia.org/T171352) (owner: 10Florianschmidtwelzow) [09:40:41] (03Merged) 10jenkins-bot: Recognize recheck with inline comments, too [integration/config] - 10https://gerrit.wikimedia.org/r/367691 (https://phabricator.wikimedia.org/T171352) (owner: 10Florianschmidtwelzow) [09:41:31] 10Continuous-Integration-Config, 10Release-Engineering-Team (Backlog), 10Patch-For-Review, 10Zuul: recheck is ignored if there are also inline comments - https://phabricator.wikimedia.org/T171352#3477360 (10hashar) 05Open>03Resolved Should be good now. [09:42:18] (03Abandoned) 10Hashar: publishing: drop postbuildscript [integration/config] - 10https://gerrit.wikimedia.org/r/363047 (owner: 10Hashar) [09:48:33] (03PS1) 10Umherirrender: [RandomSelection] Add npm job [integration/config] - 10https://gerrit.wikimedia.org/r/368152 [09:54:20] (03PS1) 10Hashar: 'recheck' on CR+2 now triggers gate-and-submit [integration/config] - 10https://gerrit.wikimedia.org/r/368154 (https://phabricator.wikimedia.org/T105474) [09:54:27] (03CR) 10jerkins-bot: [V: 04-1] 'recheck' on CR+2 now triggers gate-and-submit [integration/config] - 10https://gerrit.wikimedia.org/r/368154 (https://phabricator.wikimedia.org/T105474) (owner: 10Hashar) [09:55:05] (03Abandoned) 10Hashar: 'recheck gate' on CR+2 now triggers gate-and-submit [integration/config] - 10https://gerrit.wikimedia.org/r/227223 (https://phabricator.wikimedia.org/T105474) (owner: 10Hashar) [09:56:21] (03CR) 10Hashar: [C: 04-1] "Need:" [integration/config] - 10https://gerrit.wikimedia.org/r/368154 (https://phabricator.wikimedia.org/T105474) (owner: 10Hashar) [10:03:21] PROBLEM - Puppet errors on integration-r-lang-01 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [10:43:22] RECOVERY - Puppet errors on integration-r-lang-01 is OK: OK: Less than 1.00% above the threshold [0.0] [10:49:27] Yippee, build fixed! [10:49:28] Project selenium-MultimediaViewer » safari,beta,OS X 10.9,BrowserTests build #466: 09FIXED in 12 min: https://integration.wikimedia.org/ci/job/selenium-MultimediaViewer/BROWSER=safari,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=OS%20X%2010.9,label=BrowserTests/466/ [11:00:05] Yippee, build fixed! [11:00:06] Project selenium-MultimediaViewer » firefox,beta,Linux,BrowserTests build #466: 09FIXED in 23 min: https://integration.wikimedia.org/ci/job/selenium-MultimediaViewer/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=BrowserTests/466/ [11:01:37] 10Scap, 10MediaWiki-extensions-LocalisationUpdate: l10nupdate failing with "git pull of extensions failed" since July 19th - https://phabricator.wikimedia.org/T171711#3477619 (10Reedy) It ran fine this morning ``` 03:12 l10nupdate@tin: ResourceLoader cache refresh completed at Thu Jul 27 03:12:30 UTC 2017 (dur... [11:09:47] 10Browser-Tests-Infrastructure, 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban), 10Wikidata, and 2 others: Run Wikibase daily browser tests on Jenkins - https://phabricator.wikimedia.org/T167432#3477649 (10zeljkofilipin) [11:12:38] Yippee, build fixed! [11:12:38] Project selenium-MinervaNeue » chrome,beta,Linux,BrowserTests build #26: 09FIXED in 35 min: https://integration.wikimedia.org/ci/job/selenium-MinervaNeue/BROWSER=chrome,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=BrowserTests/26/ [11:13:35] 10Browser-Tests-Infrastructure, 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban), 10Wikidata, and 2 others: Run Wikibase daily browser tests on Jenkins - https://phabricator.wikimedia.org/T167432#3477666 (10zeljkofilipin) Created a copy of [[ https://integration.wikimedia.org/ci... [11:32:48] Yippee, build fixed! [11:32:48] Project selenium-MinervaNeue » firefox,beta,Linux,BrowserTests build #26: 09FIXED in 56 min: https://integration.wikimedia.org/ci/job/selenium-MinervaNeue/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=BrowserTests/26/ [12:11:49] (03PS3) 10XZise: [FEAT] Rewrite commit message validator [integration/commit-message-validator] - 10https://gerrit.wikimedia.org/r/367907 [12:22:38] 10Browser-Tests-Infrastructure, 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban), 10Wikidata, and 3 others: Run Wikibase daily browser tests on Jenkins - https://phabricator.wikimedia.org/T167432#3477786 (10zeljkofilipin) - Tried finding chromedriver at the machine with this [[... [12:28:39] 10Scap, 10MediaWiki-extensions-LocalisationUpdate: l10nupdate failing with "git pull of extensions failed" since July 19th - https://phabricator.wikimedia.org/T171711#3473889 (10Amire80) >>! In T171711#3477619, @Reedy wrote: > It ran fine this morning > ``` > 03:12 l10nupdate@tin: ResourceLoader cache refresh... [13:00:40] (03PS2) 10Hashar: 'recheck' on CR+2 now triggers gate-and-submit [integration/config] - 10https://gerrit.wikimedia.org/r/368154 (https://phabricator.wikimedia.org/T105474) [13:14:40] PROBLEM - Puppet errors on deployment-mathoid is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [13:26:05] 10Browser-Tests-Infrastructure, 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban), 10Wikidata, and 3 others: Run Wikibase daily browser tests on Jenkins - https://phabricator.wikimedia.org/T167432#3477931 (10zeljkofilipin) - Tried running jobs on [[ https://integration.wikimedia.... [13:32:13] 10Release-Engineering-Team (Kanban), 10Reading-Web-Backlog, 10RelatedArticles, 10MW-1.30-release-notes (WMF-deploy-2017-08-01_(1.30.0-wmf.12)), 10User-zeljkofilipin: Rewrite Related pages browser tests in Node.js - https://phabricator.wikimedia.org/T164024#3477965 (10zeljkofilipin) @Jdlrobson oops, did n... [13:34:35] 10Release-Engineering-Team (Kanban), 10RelatedArticles, 10User-zeljkofilipin: Create Jenkins job that runs RelatedArticles Selenium tests daily - https://phabricator.wikimedia.org/T171847#3477969 (10zeljkofilipin) [13:35:32] 10Release-Engineering-Team (Kanban), 10Reading-Web-Backlog, 10RelatedArticles, 10MW-1.30-release-notes (WMF-deploy-2017-08-01_(1.30.0-wmf.12)), 10User-zeljkofilipin: Rewrite Related pages browser tests in Node.js - https://phabricator.wikimedia.org/T164024#3218631 (10zeljkofilipin) [13:44:12] I know this is probably not the best place to ask, but I'm getting "remote: Unauthorized fatal: Authentication failed for 'https://gerrit.wikimedia.org/r/operations/mediawiki-config/'"after trying to submit a patch. Any ideas why? [13:44:37] Reception123 did you change your password? [13:44:44] oh wait [13:44:50] not recently, no [13:44:57] (definitely not after my account was last fixed) [13:45:03] https://gerrit.wikimedia.org/r/#/settings/http-password [13:45:09] Reception123 ^^ [13:45:14] you have to generate a password [13:45:31] note that you will have to remeber it [13:46:14] Cannot assign user name "reception123" to account 5150; name already in use. [13:46:15] oh no [13:46:18] the error just reappeared [13:46:27] It worked just fine a few hours ago [13:47:09] 10Scap, 10MediaWiki-extensions-LocalisationUpdate: l10nupdate failing with "git pull of extensions failed" since July 19th - https://phabricator.wikimedia.org/T171711#3478030 (10Reedy) Looks unrelated, stack trace attached. >>! In T171711#3477803, @Amire80 wrote: > OK, then why does the page https://din.wikip... [13:48:04] 10Gerrit, 10Release-Engineering-Team (Kanban), 10Regression, 10Upstream: Cannot log into Gerrit as of recent upgrade - https://phabricator.wikimedia.org/T152640#3478035 (10Reception123) @demon My account worked for a while, but I just tried to log in and got the same error. [13:50:55] Yippee, build fixed! [13:50:55] Project selenium-VisualEditor » firefox,beta,Linux,BrowserTests build #473: 09FIXED in 6 min 54 sec: https://integration.wikimedia.org/ci/job/selenium-VisualEditor/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=BrowserTests/473/ [13:53:42] PROBLEM - Puppet errors on deployment-mira is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [13:55:50] 10Release-Engineering-Team (Kanban), 10MobileFrontend, 10Reading-Web-Backlog, 10RelatedArticles, and 2 others: [EPIC] Port Selenium tests from Ruby to Node.js on Reading Web extensions - https://phabricator.wikimedia.org/T162256#3478047 (10zeljkofilipin) a:05zeljkofilipin>03None >>! In T162256#3466643,... [13:57:43] 10Release-Engineering-Team (Kanban), 10Reading-Web-Backlog, 10RelatedArticles, 10MW-1.30-release-notes (WMF-deploy-2017-08-01_(1.30.0-wmf.12)), 10User-zeljkofilipin: Rewrite Related pages browser tests in Node.js - https://phabricator.wikimedia.org/T164024#3478056 (10zeljkofilipin) a:05zeljkofilipin>0... [14:01:57] 10Browser-Tests-Infrastructure, 10releng-201718-q1, 10MediaWiki-General-or-Unknown, 10Epic, and 6 others: Port Selenium tests from Ruby to Node.js - https://phabricator.wikimedia.org/T139740#3478066 (10zeljkofilipin) [14:03:29] 10Browser-Tests-Infrastructure, 10Release-Engineering-Team (Kanban), 10Collaboration-Team-Triage, 10Notifications, and 3 others: Port Echo Selenium tests from Ruby to Node.js - https://phabricator.wikimedia.org/T171848#3478068 (10zeljkofilipin) [14:05:12] 10Browser-Tests-Infrastructure, 10releng-201718-q1, 10MediaWiki-General-or-Unknown, 10Epic, and 6 others: Port Selenium tests from Ruby to Node.js - https://phabricator.wikimedia.org/T139740#3478088 (10zeljkofilipin) [14:07:40] 10Browser-Tests-Infrastructure, 10releng-201718-q1, 10MediaWiki-General-or-Unknown, 10Epic, and 6 others: Port Selenium tests from Ruby to Node.js - https://phabricator.wikimedia.org/T139740#3478096 (10zeljkofilipin) [14:07:43] 10Release-Engineering-Team (Kanban), 10Math, 10Browser-Tests, 10JavaScript, and 2 others: WebdriverIO tests for Math - https://phabricator.wikimedia.org/T162455#3478095 (10zeljkofilipin) 05Open>03stalled [14:08:30] 10Browser-Tests-Infrastructure, 10releng-201718-q1, 10MediaWiki-General-or-Unknown, 10Epic, and 6 others: Port Selenium tests from Ruby to Node.js - https://phabricator.wikimedia.org/T139740#3139928 (10zeljkofilipin) [14:08:40] 10Browser-Tests-Infrastructure, 10releng-201718-q1, 10MediaWiki-General-or-Unknown, 10Epic, and 6 others: Port Selenium tests from Ruby to Node.js - https://phabricator.wikimedia.org/T139740#3141195 (10zeljkofilipin) [14:10:34] 10Browser-Tests-Infrastructure, 10releng-201718-q1, 10MediaWiki-General-or-Unknown, 10Epic, and 6 others: Port Selenium tests from Ruby to Node.js - https://phabricator.wikimedia.org/T139740#3478138 (10zeljkofilipin) [14:11:02] 10Browser-Tests-Infrastructure, 10releng-201718-q1, 10MediaWiki-General-or-Unknown, 10Epic, and 6 others: Port Selenium tests from Ruby to Node.js - https://phabricator.wikimedia.org/T139740#3146392 (10zeljkofilipin) [14:12:15] 10Browser-Tests-Infrastructure, 10releng-201718-q1, 10MediaWiki-General-or-Unknown, 10Epic, and 6 others: Port Selenium tests from Ruby to Node.js - https://phabricator.wikimedia.org/T139740#3146785 (10zeljkofilipin) [14:13:40] 10Browser-Tests-Infrastructure, 10releng-201718-q1, 10MediaWiki-General-or-Unknown, 10Epic, and 6 others: Port Selenium tests from Ruby to Node.js - https://phabricator.wikimedia.org/T139740#3478163 (10zeljkofilipin) [14:14:34] 10Release-Engineering-Team (Kanban), 10MobileFrontend, 10Reading-Web-Backlog, 10RelatedArticles, and 2 others: [EPIC] Port Selenium tests from Ruby to Node.js on Reading Web extensions - https://phabricator.wikimedia.org/T162256#3478165 (10zeljkofilipin) [14:16:37] 10Browser-Tests-Infrastructure, 10Release-Engineering-Team (Kanban), 10User-zeljkofilipin: WebdriverIO tech talk - https://phabricator.wikimedia.org/T171852#3478168 (10zeljkofilipin) [14:17:16] 10Browser-Tests-Infrastructure, 10Release-Engineering-Team (Kanban), 10User-zeljkofilipin: WebdriverIO tech talk - https://phabricator.wikimedia.org/T171852#3478168 (10zeljkofilipin) [14:17:43] 10Browser-Tests-Infrastructure, 10Release-Engineering-Team (Kanban), 10User-zeljkofilipin: WebdriverIO tech talk - https://phabricator.wikimedia.org/T171852#3478168 (10zeljkofilipin) @Jrbranaa are you in charge of tech talks now? [14:18:10] 10Browser-Tests-Infrastructure, 10releng-201718-q1, 10MediaWiki-General-or-Unknown, 10Epic, and 6 others: Port Selenium tests from Ruby to Node.js - https://phabricator.wikimedia.org/T139740#3478191 (10zeljkofilipin) [14:19:27] 10Browser-Tests-Infrastructure, 10Release-Engineering-Team (Kanban), 10User-zeljkofilipin: WebdriverIO tech talk - https://phabricator.wikimedia.org/T171852#3478168 (10zeljkofilipin) p:05Triage>03Normal [14:19:40] RECOVERY - Puppet errors on deployment-mathoid is OK: OK: Less than 1.00% above the threshold [0.0] [14:23:45] 10Browser-Tests-Infrastructure, 10releng-201718-q1, 10MediaWiki-General-or-Unknown, 10Epic, and 6 others: Port Selenium tests from Ruby to Node.js - https://phabricator.wikimedia.org/T139740#3478216 (10zeljkofilipin) [14:27:45] 10Browser-Tests-Infrastructure, 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban), 10Wikidata, and 3 others: Run Wikibase daily browser tests on Jenkins - https://phabricator.wikimedia.org/T167432#3478230 (10zeljkofilipin) The first test that runs fine, thanks to @hashar! https:... [14:29:06] 10Browser-Tests-Infrastructure, 10releng-201718-q1, 10MediaWiki-General-or-Unknown, 10Epic, and 6 others: Port Selenium tests from Ruby to Node.js - https://phabricator.wikimedia.org/T139740#3478244 (10zeljkofilipin) [14:30:19] (03CR) 10Hashar: [C: 032] [RandomSelection] Add npm job [integration/config] - 10https://gerrit.wikimedia.org/r/368152 (owner: 10Umherirrender) [14:31:13] (03Merged) 10jenkins-bot: [RandomSelection] Add npm job [integration/config] - 10https://gerrit.wikimedia.org/r/368152 (owner: 10Umherirrender) [14:33:43] RECOVERY - Puppet errors on deployment-mira is OK: OK: Less than 1.00% above the threshold [0.0] [14:40:35] 10Release-Engineering-Team (Kanban), 10Operations, 10Patch-For-Review, 10Security-General: setup releases1001.eqiad.wmnet (was: setup mwreleases1001) - https://phabricator.wikimedia.org/T164030#3478267 (10Dzahn) a:05demon>03Dzahn [14:52:29] 10Release-Engineering-Team (Watching / External), 10Scap, 10ORES, 10Scoring-platform-team-Backlog: Simplify git-fat support for pulling from both production and labs - https://phabricator.wikimedia.org/T171758#3475312 (10Halfak) 05Open>03stalled p:05Normal>03High [14:52:32] 10Release-Engineering-Team (Watching / External), 10Scap, 10ORES, 10Operations, 10Scoring-platform-team-Backlog: ORES should use git-fat for wheel deployments - https://phabricator.wikimedia.org/T171619#3478291 (10Halfak) [14:52:57] 10Release-Engineering-Team (Watching / External), 10Scap, 10ORES, 10Operations, 10Scoring-platform-team-Backlog: ORES should use git-fat for wheel deployments - https://phabricator.wikimedia.org/T171619#3470967 (10Halfak) [14:53:01] 10Release-Engineering-Team (Watching / External), 10Scap, 10ORES, 10Scoring-platform-team-Backlog: Simplify git-fat support for pulling from both production and labs - https://phabricator.wikimedia.org/T171758#3475312 (10Halfak) 05stalled>03Resolved p:05High>03Normal [14:53:11] 10Release-Engineering-Team (Watching / External), 10Scap, 10ORES, 10Scoring-platform-team: Simplify git-fat support for pulling from both production and labs - https://phabricator.wikimedia.org/T171758#3475312 (10Halfak) [14:53:18] 10Release-Engineering-Team (Watching / External), 10Scap, 10ORES, 10Operations, 10Scoring-platform-team-Backlog: ORES should use git-fat for wheel deployments - https://phabricator.wikimedia.org/T171619#3470967 (10Halfak) [14:53:21] 10Release-Engineering-Team (Watching / External), 10Scap, 10ORES, 10Scoring-platform-team: Simplify git-fat support for pulling from both production and labs - https://phabricator.wikimedia.org/T171758#3475312 (10Halfak) 05Resolved>03Open [14:54:05] 10Release-Engineering-Team (Watching / External), 10Scap, 10ORES, 10Operations, 10Scoring-platform-team-Backlog: ORES should use git-fat for binaries - https://phabricator.wikimedia.org/T171619#3470967 (10Halfak) [14:54:28] 10Release-Engineering-Team (Watching / External), 10Scap, 10ORES, 10Operations, 10Scoring-platform-team-Backlog: ORES should use git-fat for binaries - https://phabricator.wikimedia.org/T171619#3470967 (10Halfak) 05Open>03stalled p:05Normal>03High [15:05:36] 10Browser-Tests-Infrastructure, 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban), 10Wikidata, and 3 others: Run Wikibase daily browser tests on Jenkins - https://phabricator.wikimedia.org/T167432#3478331 (10zeljkofilipin) >>! In T167432#3469820, @Aleksey_WMDE wrote: > It's not e... [15:12:10] uh oh https://phabricator.wikimedia.org/T152640#3478035 [15:13:07] RainbowSprinkles ^^ [15:13:32] Reception123 looks like your missing the gerrit: prefix [15:13:40] Again? [15:13:43] we should re add it and do a reindex to see if that fixes it [15:13:52] Ok, sounds good [15:14:03] Reception123 the error your describing means that the gerrit: prefix is missing from your account [15:14:03] so then it probably got removed again when I logged in? [15:14:11] there you go, succesfully connected you guys :) [15:14:16] has to go for an hour [15:14:22] then will do releases.wm.org [15:15:00] Reception123 i have therys but nothing to confirm [15:15:14] my thiery is that the index is causing the db to return to it's old self [15:15:24] but upstream are saying that does not happen [15:15:56] Ok. Then we should wait for RainbowSprinkles to confirm if that is the case, and do the reindex [15:15:56] mutante: Ack, thx [15:15:58] :) [15:16:47] Reception123 reindex can be done online :) [15:16:47] hi folks! any chance you could tell me which version of npm is running in https://integration.wikimedia.org/ci/job/phabricator-jessie-commits ? [15:16:54] npm 2 [15:17:09] ohhh I see [15:17:17] paladox: Ok, but the gerrit: prefix still needs to be added anyway so we still need RainbowSprinkles :) [15:17:28] that might be the origin of the problems [15:17:51] fdans: Are you sure npm itself isn't the origin of the problem? ;-) [15:17:54] * RainbowSprinkles goes back to his coffee [15:18:12] oh npm is the origin of aaaall problems [15:18:22] Reception123: yep, what i meant was the reindex can be done online still requires chad to run that. But because the problem has come back for you the only thing we did not do was the reindex. [15:18:34] but in this particular scope of problems, it might be the old version RainbowSprinkles :) [15:18:35] fdans: RIGHT?! [15:18:44] Ok then [15:20:51] RainbowSprinkles: this is an absolute noob question buuuut... any chance we could upgrade npm? 😬 [15:21:22] paladox, Reception123: Added the external_id entry and forced a reindex [15:21:33] thanks RainbowSprinkles :) [15:21:36] fdans: I'm the wrong guy to ask...hashar? [15:21:47] James_F asked that too [15:21:48] RainbowSprinkles: works, thanks again [15:21:50] there's a task [15:22:14] https://phabricator.wikimedia.org/T161861 [15:22:45] 10Continuous-Integration-Infrastructure: Jobs with Node 6 should also have npm 3 - https://phabricator.wikimedia.org/T161861#3478420 (10Paladox) We could do npm install -g npm@3 or npm install -g npm before the node image is built. [15:24:51] RainbowSprinkles 2.14 should deffitly fix this problem, it reindexes accounts after doing certain things now (forgot what they were but related to external id) [15:28:07] 10Gerrit, 10Release-Engineering-Team (Kanban), 10Regression, 10Upstream: Cannot log into Gerrit as of recent upgrade - https://phabricator.wikimedia.org/T152640#3478449 (10Paladox) [15:28:40] paladox: this is probably a very obvious mistake I made, but "git clone ssh:///Reception123@gerrit.wikimedia.org:29418/operations/mediawiki-config" gives "ssh: connect to host port 22: Connection refused" [15:28:51] uh [15:29:02] oh wait [15:29:11] why is it trying to connect to port 22? [15:29:39] git clone git clone ssh://Reception123@gerrit.wikimedia.org:29418/operations/mediawiki-config [15:29:45] git clone ssh://Reception123@gerrit.wikimedia.org:29418/operations/mediawiki-config [15:30:54] hmm Permission denied (publickey). fatal: Could not read from remote repository. [15:31:05] I added my key to gerrit [15:31:44] Use https ;-) [15:32:18] I did that before, and even after providing the HTTP password set in gerrit I got "fatal: Authentication failed for 'https://gerrit.wikimedia.org/r/operations/mediawiki-config/" [15:32:37] use lower case [15:32:41] username [15:33:45] same thing :( [15:35:11] hmm [15:35:24] try your ldap password [15:35:55] so then I'll try the https method again [15:36:50] LDAP password won't work for https. [15:36:50] yep [15:36:56] (I don't have that enabled on purpose) [15:37:11] https://github.com/wikimedia/puppet/blob/production/modules/gerrit/templates/gerrit.config.erb#L2 [15:37:13] well for SSH I don't get asked for my password at all [15:37:24] nope you need a ssh pub key for that [15:37:48] I have one, and also added it to Gerrit [15:38:28] ok [15:38:32] is it the correct one? [15:38:44] what format is it? [15:38:50] rsa? [15:39:07] yes RSA [15:39:25] it is correct, matches my local one [15:39:45] ok [15:39:49] hmm [15:42:27] 10Release-Engineering-Team (Kanban), 10Reading-Web-Backlog, 10RelatedArticles, 10User-zeljkofilipin: Create Jenkins job that runs RelatedArticles Selenium tests daily - https://phabricator.wikimedia.org/T171847#3478502 (10Jdlrobson) [15:42:33] RainbowSprinkles actually that's not true, it's ldap [15:42:36] we did not set https://gerrit-review.googlesource.com/Documentation/config-gerrit.html#auth.gitBasicAuthPolicy [15:43:15] which was introduced in a newer gerrit 2.13.* release then what we were running at the time [15:43:20] Reception123 try your ldap password for http :) [15:43:23] paladox: it worked on https now [15:43:29] oh [15:43:34] it an uppercase/lowercase issue [15:43:37] how did you get it to work? [15:43:40] username? [15:43:41] thanks for your help, and sorry for bothering [15:43:42] yeah.. [15:43:45] ah [15:43:46] ha [15:44:17] we need to set https://gerrit-review.googlesource.com/Documentation/config-gerrit.html#auth.gitBasicAuthPolicy RainbowSprinkles [15:44:27] Reception123 ^^ [15:44:46] * paladox submits patches [15:44:57] Yeah.. that would be better than only being able to use the HTTP pass [15:45:34] 10Scap, 10Analytics, 10EventBus: eventlogging-service-eventbus scap deployments should depool/pool during deployment - https://phabricator.wikimedia.org/T171506#3478506 (10fdans) p:05Triage>03Normal [15:47:04] (03CR) 10Florianschmidtwelzow: "Thanks for your +1 @Krinkle and merging @hashar! :)" [integration/config] - 10https://gerrit.wikimedia.org/r/367691 (https://phabricator.wikimedia.org/T171352) (owner: 10Florianschmidtwelzow) [15:51:24] 10Release-Engineering-Team (Kanban), 10RelatedArticles, 10Reading-Web-Backlog (Tracking), 10User-zeljkofilipin: Create Jenkins job that runs RelatedArticles Selenium tests daily - https://phabricator.wikimedia.org/T171847#3478541 (10Jdlrobson) [15:51:33] RainbowSprinkles Reception123: https://gerrit.wikimedia.org/r/#/c/368196/ [15:52:07] How come no one else had this issue before me? [15:52:40] not sure [15:53:48] RainbowSprinkles do you want me to set gitBasicAuthPolicy to HTTP? [15:58:04] Reception123 also to let you know, you can create changes through the inline edit :) [15:58:53] I know but I wanted to redo it because of another mistake as well, so it was easier to just redo the whole thing [16:15:07] 10Browser-Tests-Infrastructure, 10Release-Engineering-Team (Kanban), 10User-zeljkofilipin: WebdriverIO tech talk - https://phabricator.wikimedia.org/T171852#3478707 (10Jrbranaa) @zeljkofilipin Rachel is still setting them up for us, by I offered to get them going and coordinating some topics. When do you wa... [16:28:24] 10Continuous-Integration-Config, 10Release-Engineering-Team, 10Jenkins, 10Reading-Web-Backlog (Tracking): Jenkins should run `npm test` on core - https://phabricator.wikimedia.org/T171866#3478741 (10Jdlrobson) [16:28:49] 10Continuous-Integration-Config, 10Release-Engineering-Team, 10Jenkins, 10Reading-Web-Backlog (Tracking): Jenkins should run `npm test` on core - https://phabricator.wikimedia.org/T171866#3478753 (10Jdlrobson) [16:56:37] that https://gerrit-review.googlesource.com/#/c/113412/ will make it easy to find changes that are reverts :) [17:11:32] Project selenium-MinervaNeue » chrome,beta,Linux,BrowserTests build #27: 04FAILURE in 27 min: https://integration.wikimedia.org/ci/job/selenium-MinervaNeue/BROWSER=chrome,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=BrowserTests/27/ [17:19:09] Project selenium-MinervaNeue » firefox,beta,Linux,BrowserTests build #27: 04FAILURE in 35 min: https://integration.wikimedia.org/ci/job/selenium-MinervaNeue/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=BrowserTests/27/ [17:24:35] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team (Kanban), 10Wikidata: beta-update-databases-eqiad times out on wikidatawiki (maintenance update) - https://phabricator.wikimedia.org/T167981#3478933 (10greg) [17:24:40] 10Beta-Cluster-Infrastructure, 10MediaWiki-extensions-WikibaseRepository, 10Wikidata: rebuildTermSqlIndex.php causing beta-update-databases-eqiad to timeout and abort - https://phabricator.wikimedia.org/T168036#3478931 (10greg) 05Open>03Resolved Reopen if not :) [17:29:47] 10Scap, 10MediaWiki-extensions-LocalisationUpdate: l10nupdate failing with "git pull of extensions failed" since July 19th - https://phabricator.wikimedia.org/T171711#3478953 (10greg) >>! In T171711#3477126, @Amire80 wrote: > Isn't there generic monitoring for errors? Not sure exactly what you mean, but logst... [17:34:36] thcipriani: is this resolved? https://phabricator.wikimedia.org/T170597 [17:34:42] I can't check logstash [17:35:04] * thcipriani checks [17:39:07] Amir1: looks like it is, I no longer see the message in logstash, I resolved the task. Thank you for the ping. [17:39:38] thcipriani: thanks for noticing :) [17:40:30] :D [18:32:22] PROBLEM - Puppet errors on deployment-conf03 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [18:36:20] 10Continuous-Integration-Config, 10Release-Engineering-Team, 10Jenkins, 10Reading-Web-Backlog (Tracking): Jenkins should run `npm test` on core - https://phabricator.wikimedia.org/T171866#3478741 (10hashar) The job is `mediawiki-core-npm-node-6-jessie` but it is optimized out to only trigger on patch that... [18:39:08] 10Continuous-Integration-Config, 10Wikipedia-Android-App-Backlog, 10Patch-For-Review: Don't re-download the Android SDK for every (non-periodic) CI job - https://phabricator.wikimedia.org/T171811#3479222 (10Mholloway) From what I've observed this knocks about a minute off the apps-android-wikipedia-test time... [18:56:51] (03PS1) 10Mholloway: Provide Android SDK location as an argument to non-periodic test scripts [integration/config] - 10https://gerrit.wikimedia.org/r/368238 (https://phabricator.wikimedia.org/T171811) [18:59:33] RainbowSprinkles: starting with finalizing releases1001... [18:59:41] Wheeeee :) [18:59:48] adding varnish misc director "releases" [18:59:51] step 1 [19:01:08] 10Continuous-Integration-Infrastructure: Use phpcs.xml of core in mediawiki-core-phpcs-jessie - https://phabricator.wikimedia.org/T170159#3479284 (10Fomafix) Seams to be fixed. https://gerrit.wikimedia.org/r/#/c/251312/36 reports no error anymore. [19:12:20] RECOVERY - Puppet errors on deployment-conf03 is OK: OK: Less than 1.00% above the threshold [0.0] [19:42:43] 10Release-Engineering-Team (Kanban), 10Release, 10Train Deployments: 1.30.0-wmf.11 deployment blockers - https://phabricator.wikimedia.org/T168051#3479486 (10thcipriani) [19:45:05] switching the backend of releases.wikimedia.org [19:45:17] puppet running on cache::misc right now [19:45:23] all files were rsynced [19:45:40] curl localhost on releases1001 was just fine [19:46:10] since there is a new director "releases" in cache::misc, it will be ready for active/acttive in the future - to add releases2001 [19:51:54] 10Release-Engineering-Team (Kanban), 10CirrusSearch, 10Discovery, 10Discovery-Search, and 2 others: Figure out why browser tests can't create suggestion box - https://phabricator.wikimedia.org/T162966#3479503 (10EBernhardson) I've just tried installing a newer phantomjs, specifically 2.1.7, but it runs int... [19:55:36] well.. done and unspectacular.. just didn't test the upload of a new deb because i dont really know which one to use for testing and dont want to mess up reprepro with "test" fake package [19:56:08] maybe we can just do it when we actually have a release and you can make me fix it if an issue come up [20:07:30] 2.15 gains a new permission backend too. [20:09:25] removing old role from bromine [20:09:49] (which will affect shell access) [20:10:06] or not, heh, checking that part [20:27:25] 10MediaWiki-Codesniffer: Check @covers against autoloader with codesniffer - https://phabricator.wikimedia.org/T171899#3479635 (10Umherirrender) [21:03:51] 10Release-Engineering-Team (Kanban), 10Release, 10Train Deployments: 1.30.0-wmf.11 deployment blockers - https://phabricator.wikimedia.org/T168051#3479767 (10MacFan4000) 05Open>03Resolved wmf.11 is successfully deployed [21:15:20] 10Release-Engineering-Team (Watching / External), 10Phlogiston, 10User-KSmith: Adjust phlogiston configuration for Release Engineering - https://phabricator.wikimedia.org/T170359#3479796 (10ksmith) [21:30:17] is there a way for me, as a mere mortal, to create a phabricator tag? [21:34:05] PROBLEM - Puppet errors on deployment-kafka01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [21:34:19] SMalyshev: afaik Andre__ wants you to request them to keep the names consistent [21:34:44] ok, who I should tag on that request? [21:35:46] PROBLEM - Puppet errors on deployment-etcd-01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [21:36:28] SMalyshev: i was trying to find this https://www.mediawiki.org/wiki/Phabricator/Creating_and_renaming_projects#Creating_new_projects [21:36:46] "Creating projects requires being a member of Project-Admins. If you would like to create (sub)projects, add a comment in phab:T706 to become a member." [21:36:51] T706: Requests for addition to the #acl*Project-Admins group (in comments) - https://phabricator.wikimedia.org/T706 [21:36:51] PROBLEM - Puppet errors on deployment-pdf01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [21:37:16] Tag: If you want to create a Tag project, create a task to allow discussion of the proposal. (Example: T127007) [21:37:17] T127007: Create a generic #Goal tag - https://phabricator.wikimedia.org/T127007 [21:37:32] SMalyshev: i would copy the people from https://phabricator.wikimedia.org/T127007 because that's the official example ticket [21:37:43] or you can request to become project admin and then create them yourself in the future [21:37:49] aha, cool, thanks [21:37:51] yw [21:40:11] it says there the tags are for cross-component things. What if I'd like to have tag that applies to only one component (i.e. tasks that relate to bugs in specific sub-component)? [21:40:27] is it ok to create tags for those too? [21:48:49] SMalyshev: I don't think there's a specific ruling on the matter, but as long as the name is clear, something like "#search-goals" or whatever, not just "goals" [21:49:06] greg-g: ok, thanks! [22:29:29] Yippee, build fixed! [22:29:29] Project selenium-MinervaNeue » firefox,beta,Linux,BrowserTests build #29: 09FIXED in 38 min: https://integration.wikimedia.org/ci/job/selenium-MinervaNeue/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=BrowserTests/29/ [22:44:19] PROBLEM - Puppet errors on deployment-mcs01 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [22:53:14] 10Release-Engineering-Team (Kanban), 10Operations: setup releases2001.codfw.wmnet - https://phabricator.wikimedia.org/T171917#3480023 (10Dzahn) [22:53:49] 10Release-Engineering-Team (Watching / External), 10Operations: Provide cross-dc redundancy (active-active or active-passive) to all important misc services - https://phabricator.wikimedia.org/T156937#2990470 (10Dzahn) [22:53:51] 10Release-Engineering-Team (Kanban), 10Operations: setup releases2001.codfw.wmnet - https://phabricator.wikimedia.org/T171917#3480023 (10Dzahn) [22:54:57] 10Release-Engineering-Team (Watching / External), 10Operations: Provide cross-dc redundancy (active-active or active-passive) to all important misc services - https://phabricator.wikimedia.org/T156937#2990470 (10Dzahn) [22:56:15] Yippee, build fixed! [22:56:15] Project selenium-MinervaNeue » chrome,beta,Linux,BrowserTests build #30: 09FIXED in 23 min: https://integration.wikimedia.org/ci/job/selenium-MinervaNeue/BROWSER=chrome,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=BrowserTests/30/ [23:15:15] 10Release-Engineering-Team (Kanban), 10Operations, 10Patch-For-Review: setup releases2001.codfw.wmnet - https://phabricator.wikimedia.org/T171917#3480100 (10Dzahn) [23:15:38] 10Release-Engineering-Team (Kanban), 10Operations, 10vm-requests, 10Patch-For-Review: setup releases2001.codfw.wmnet - https://phabricator.wikimedia.org/T171917#3480023 (10Dzahn) [23:19:18] RECOVERY - Puppet errors on deployment-mcs01 is OK: OK: Less than 1.00% above the threshold [0.0] [23:40:45] 10Release-Engineering-Team, 10Reading-Web-Backlog, 10Wikimedia-Site-requests, 10Design (RW-Design-Debt), and 2 others: Deploy common Wordmarks - https://phabricator.wikimedia.org/T171556#3480153 (10Nirzar) 05Open>03Resolved a:03Nirzar Compared with desktop logos. {meme, src=votecat, above="looks "... [23:42:53] 10Release-Engineering-Team (Kanban), 10Operations, 10vm-requests, 10Patch-For-Review: setup releases2001.codfw.wmnet - https://phabricator.wikimedia.org/T171917#3480158 (10Dzahn) releases2001.codfw.wmnet has address 10.192.16.174 releases2001.codfw.wmnet has IPv6 address 2620:0:860:102:10:192:16:174 [23:57:48] 10Release-Engineering-Team, 10MediaWiki-JobRunner, 10Performance-Team: Investigate 30x increase in Jobrunner errors - https://phabricator.wikimedia.org/T171371#3462630 (10Krinkle)