[01:24:49] 10Continuous-Integration-Scaling, 10Cloud-Services, 10Patch-For-Review: Nodepool can not spawn instances anymore on wmflabs - https://phabricator.wikimedia.org/T135631#3504997 (10Liuxinyu970226) [01:25:11] 10Continuous-Integration-Scaling, 10Cloud-VPS, 10Patch-For-Review: Nodepool can not spawn instances anymore on wmflabs - https://phabricator.wikimedia.org/T135631#3505003 (10Liuxinyu970226) [01:25:31] 10Continuous-Integration-Scaling, 10Cloud-Services, 10Patch-For-Review: Nodepool can not spawn instances anymore on wmflabs - https://phabricator.wikimedia.org/T135631#2305151 (10Liuxinyu970226) [04:01:43] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban), 10Analytics-Kanban, 10Analytics-Wikistats: Fix Wikistats build in Jenkins - https://phabricator.wikimedia.org/T171599#3505034 (10Nuria) [06:40:49] PROBLEM - Mediawiki Error Rate on graphite-labs is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [10.0] [07:40:51] PROBLEM - Mediawiki Error Rate on graphite-labs is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [10.0] [07:49:42] 10Release-Engineering-Team (Kanban), 10Phabricator, 10Availability, 10Patch-For-Review, 10WorkType-NewFunctionality: Deploy phabricator to phab2001.codfw.wmnet - https://phabricator.wikimedia.org/T137928#3505170 (10jcrespo) can I ask which database is phab2001 connecting to right now? [08:24:45] 10Release-Engineering-Team (Kanban), 10Phabricator, 10Availability, 10Patch-For-Review, 10WorkType-NewFunctionality: Deploy phabricator to phab2001.codfw.wmnet - https://phabricator.wikimedia.org/T137928#3505193 (10TerraCodes) [08:50:49] RECOVERY - Mediawiki Error Rate on graphite-labs is OK: OK: Less than 1.00% above the threshold [1.0] [10:13:28] 10Release-Engineering-Team (Kanban), 10Phabricator, 10Availability, 10Patch-For-Review, 10WorkType-NewFunctionality: Deploy phabricator to phab2001.codfw.wmnet - https://phabricator.wikimedia.org/T137928#3505467 (10Paladox) >>! In T137928#3505170, @jcrespo wrote: > can I ask which database is phab2001 co... [11:21:14] PROBLEM - Puppet errors on deployment-sca02 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [12:22:41] Project selenium-GettingStarted » firefox,beta,Linux,BrowserTests build #487: 04FAILURE in 40 sec: https://integration.wikimedia.org/ci/job/selenium-GettingStarted/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=BrowserTests/487/ [13:12:28] zeljkof: Hey, there is no one for SWAT, I have the rights but I don't know if that's okay to do so [13:12:42] greg-g: Reedy ragesoss twentyafterfour ^ [13:12:57] Sorry, meant to ping RainbowSprinkles [13:14:28] Amir1: hashar and I are on vacation [13:15:10] zeljkof: have fun :) I'm not saying you do, I'm saying if that's okay if I do [13:15:16] maybe we can allow him to deploy? [13:15:21] If nobody claimed the swat, and you know how to deploy and you are able... [13:15:41] [15:15] _joe_ yes, SWAT windows should be done by official swatters only without explicit permission or UBN! tickets [13:15:41] Thanks! [13:16:24] Not sure if there is a list of official swatters :) [13:18:00] Amir1: To be explicit, I can not allow or deny you to deploy [13:18:31] Of nobody here complains... [13:18:40] *if... [13:19:07] zeljkof: looks like a good topic for ops@lists [13:19:14] what to do if/when... [13:19:44] zeljkof: well, I can't deploy today at all, forgot the yubikey [13:20:07] facepalm [13:20:14] Problem solved! :D [13:20:43] Oh, just noticed we are in releng channel [13:23:50] 10Release-Engineering-Team (Kanban), 10Phabricator, 10Availability, 10Patch-For-Review, 10WorkType-NewFunctionality: Deploy phabricator to phab2001.codfw.wmnet - https://phabricator.wikimedia.org/T137928#3505775 (10Dzahn) The answer should be done, as the phd service isn't running there and nothing chang... [13:29:04] and thcipriani|afk is really afk? :) [13:40:51] 10Release-Engineering-Team (Kanban), 10Phabricator, 10Availability, 10Patch-For-Review, 10WorkType-NewFunctionality: Deploy phabricator to phab2001.codfw.wmnet - https://phabricator.wikimedia.org/T137928#3505799 (10jcrespo) @Dzahn, thanks- I saw paladox comment: > Is codfw ment to be slow? It feels slow... [13:44:49] PROBLEM - Parsoid on deployment-parsoid09 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:54:40] RECOVERY - Parsoid on deployment-parsoid09 is OK: HTTP OK: HTTP/1.1 200 OK - 1051 bytes in 4.383 second response time [14:03:36] PROBLEM - Puppet errors on integration-r-lang-01 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [14:28:19] 10Release-Engineering-Team (Kanban), 10Phabricator, 10Availability, 10Patch-For-Review, 10WorkType-NewFunctionality: Deploy phabricator to phab2001.codfw.wmnet - https://phabricator.wikimedia.org/T137928#3505972 (10Dzahn) Yea, so the service isn't up, but the config is indeed like this: hieradata/role/e... [14:33:39] Project selenium-WikiLove » firefox,beta,Linux,BrowserTests build #478: 04FAILURE in 1 min 38 sec: https://integration.wikimedia.org/ci/job/selenium-WikiLove/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=BrowserTests/478/ [14:36:50] 10Release-Engineering-Team (Kanban), 10Phabricator, 10Availability, 10Patch-For-Review, 10WorkType-NewFunctionality: Deploy phabricator to phab2001.codfw.wmnet - https://phabricator.wikimedia.org/T137928#3505986 (10Dzahn) Just to make sure, I checked with `tcpdump port 3306` and there was nothing, even w... [14:38:03] 10Release-Engineering-Team (Kanban), 10Phabricator, 10Availability, 10Patch-For-Review, 10WorkType-NewFunctionality: Deploy phabricator to phab2001.codfw.wmnet - https://phabricator.wikimedia.org/T137928#3505993 (10jcrespo) I think we added m3-master.codfw.wmnet not a long time ago (maybe it was another... [14:42:59] 10Release-Engineering-Team (Kanban), 10Phabricator, 10Availability, 10Patch-For-Review, 10WorkType-NewFunctionality: Deploy phabricator to phab2001.codfw.wmnet - https://phabricator.wikimedia.org/T137928#3506024 (10Dzahn) Thanks! @jcrespo. Unfortunately i was wrong, there would have been cross-dc querie... [14:43:38] RECOVERY - Puppet errors on integration-r-lang-01 is OK: OK: Less than 1.00% above the threshold [0.0] [14:45:56] 10Release-Engineering-Team (Watching / External), 10Operations, 10Wikimania-Hackathon-2017-Organization: Wikimania needs hosting on a server for onsite conference guide - https://phabricator.wikimedia.org/T172217#3506028 (10Dzahn) @Antoine2711 Is this working for you? Any issues? I will be in travel to Wikim... [15:11:21] Project mediawiki-core-code-coverage build #2931: 15ABORTED in 11 min: https://integration.wikimedia.org/ci/job/mediawiki-core-code-coverage/2931/ [15:11:53] !log restarting jenkins for plugin update [15:11:57] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [16:15:28] (03PS1) 10Umherirrender: [CategoryWatch] Add npm job [integration/config] - 10https://gerrit.wikimedia.org/r/370492 [16:18:23] 10Browser-Tests-Infrastructure, 10Release-Engineering-Team (Next), 10MinervaNeue, 10Readers-Web-Backlog, and 4 others: MinervaNeue browser test are flaking (waiting for {:class=>"mw-notification", :tag_name=>"div"} to become present ) - https://phabricator.wikimedia.org/T170890#3446357 (10Jdlrobson) [16:23:20] (03CR) 10Hashar: [C: 032] Archive SemanticUpdateOnPurge [integration/config] - 10https://gerrit.wikimedia.org/r/370148 (https://phabricator.wikimedia.org/T172294) (owner: 10Florianschmidtwelzow) [16:24:11] (03PS1) 10Umherirrender: [MassMessageEmail] Add npm job [integration/config] - 10https://gerrit.wikimedia.org/r/370495 [16:24:15] (03Merged) 10jenkins-bot: Archive SemanticUpdateOnPurge [integration/config] - 10https://gerrit.wikimedia.org/r/370148 (https://phabricator.wikimedia.org/T172294) (owner: 10Florianschmidtwelzow) [16:24:25] (03CR) 10Hashar: [C: 032] [CategoryWatch] Add npm job [integration/config] - 10https://gerrit.wikimedia.org/r/370492 (owner: 10Umherirrender) [16:25:22] (03Merged) 10jenkins-bot: [CategoryWatch] Add npm job [integration/config] - 10https://gerrit.wikimedia.org/r/370492 (owner: 10Umherirrender) [16:27:14] (03CR) 10Hashar: [C: 032] [MassMessageEmail] Add npm job [integration/config] - 10https://gerrit.wikimedia.org/r/370495 (owner: 10Umherirrender) [16:27:21] 10Release-Engineering-Team (Kanban), 10Release, 10Train Deployments: 1.30.0-wmf.13 deployment blockers - https://phabricator.wikimedia.org/T170631#3506510 (10mmodell) [16:28:54] (03Merged) 10jenkins-bot: [MassMessageEmail] Add npm job [integration/config] - 10https://gerrit.wikimedia.org/r/370495 (owner: 10Umherirrender) [16:39:13] 10Deployment-Systems, 10Release-Engineering-Team (Kanban): Automate the recurring management of wikitech:Deployments and phab:#train_deployments - https://phabricator.wikimedia.org/T114488#3506570 (10mmodell) p:05High>03Normal [16:42:34] 10Release-Engineering-Team (Kanban), 10Release, 10Train Deployments: 1.30.0-wmf.13 deployment blockers - https://phabricator.wikimedia.org/T170631#3506587 (10mmodell) [16:45:17] 10Release-Engineering-Team (Watching / External), 10Pywikibot-General: Share Appveyor account credentials with Release Engineering - https://phabricator.wikimedia.org/T104306#3506593 (10greg) [16:45:20] 10Release-Engineering-Team (Kanban), 10User-zeljkofilipin: Use pwstore (a shared gpg-encrypted password store) for Release Engineering related passwords - https://phabricator.wikimedia.org/T139093#3506591 (10greg) 05Open>03Resolved a:03thcipriani [16:46:10] 10Release-Engineering-Team (Kanban), 10Operations, 10Phabricator, 10Traffic: Verify that the codfw lvs is configured correctly for Phabricator - https://phabricator.wikimedia.org/T168699#3506599 (10mmodell) phab2001 web works, git-ssh still unknown. [16:48:23] 10Release-Engineering-Team (Kanban), 10Release, 10Train Deployments: 1.30.0-wmf.13 deployment blockers - https://phabricator.wikimedia.org/T170631#3506605 (10mmodell) [16:49:40] 10Release-Engineering-Team (Kanban), 10Release, 10Train Deployments: 1.30.0-wmf.13 deployment blockers - https://phabricator.wikimedia.org/T170631#3437587 (10mmodell) The situation isn't looking good for wmf.13, we have a ton of carry-over issues from the past `3 weeks. [17:01:53] 10Release-Engineering-Team (Kanban), 10Phabricator, 10Availability, 10Patch-For-Review, 10WorkType-NewFunctionality: Deploy phabricator to phab2001.codfw.wmnet - https://phabricator.wikimedia.org/T137928#3506667 (10Dzahn) >>! In T137928#3505993, @jcrespo wrote: > I think we added m3-master.codfw.wmnet no... [17:14:34] 10Browser-Tests-Infrastructure, 10Release-Engineering-Team (Next), 10MinervaNeue, 10Readers-Web-Backlog, and 5 others: MinervaNeue browser test are flaking (waiting for {:class=>"mw-notification", :tag_name=>"div"} to become present ) - https://phabricator.wikimedia.org/T170890#3506707 (10phuedx) [17:15:07] 10Browser-Tests-Infrastructure, 10Release-Engineering-Team (Next), 10MinervaNeue, 10Readers-Web-Backlog, and 5 others: [4 hrs] MinervaNeue browser test are flaking (waiting for {:class=>"mw-notification", :tag_name=>"div"} to become present ) - https://phabricator.wikimedia.org/T170890#3446357 (10phuedx) [17:23:58] o/ o/ o/ hey all. we have a temporary github repo and a permanent (currently empty) gerrit repo. is there someone that can pull all the commits from the temporary repo to the permanent one? we don't have permissions to push directly to master. T172702 [17:24:00] T172702: Merge github repo into gerrit repo - https://phabricator.wikimedia.org/T172702 [17:24:08] temporary repo is: github.com/niedzielski/marvin [17:24:26] permanent repo: https://gerrit.wikimedia.org/r/marvin [17:33:31] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban), 10Patch-For-Review: Jenkins: Assert no PHP errors (notices, warnings) were raised or exceptions were thrown - https://phabricator.wikimedia.org/T50002#3506739 (10greg) Tyler will try to take a look at this while Antoine is out. [17:39:17] 10Beta-Cluster-Infrastructure, 10Multimedia, 10Wikimedia-Extension-setup, 10Category, 10FileAnnotations (Beta Cluster Release): Release FileAnnotations on the Beta Cluster - https://phabricator.wikimedia.org/T144302#3506765 (10MBinder_WMF) [17:57:21] 10Release-Engineering-Team (Watching / External), 10Pywikibot-General: Share Appveyor account credentials with Release Engineering - https://phabricator.wikimedia.org/T104306#3507164 (10greg) >>! In T104306#2540561, @greg wrote: > We (RelEng) can hold a copy of the password for safe-keeping, if still desired.... [18:19:04] T171940 sounds interesting [18:19:04] T171940: Create a Gadget to easily add/remove/modify patches for SWAT at wikitech:Deployments - https://phabricator.wikimedia.org/T171940 [18:24:00] https://phabricator.wikimedia.org/T172718 - someone should probably remove the random spam/files added by Discovery119 and their account should be disabled [18:28:51] disabled [18:32:27] got all of their comments removed, didn't delete files (in a meeting) [18:56:16] any gerrit people arround? jenkins keeps approving https://gerrit.wikimedia.org/r/#/c/370497/ against mediawiki-config with gate and submit, but not merging [18:56:46] doh, nevermind it worked this time :) [19:14:02] bd808 greg-g: hey there. we're trying to migrate a project from github to gerrit. basically, move the commits from github to the currently empty gerrit repo. can you direct me to a good person to help with that? the ticket is: T172702. the repos are: https://github.com/niedzielski/marvin https://gerrit.wikimedia.org/r/marvin [19:14:03] T172702: Merge github repo into gerrit repo - https://phabricator.wikimedia.org/T172702 [19:15:35] niedzielski we could mirror the repo [19:16:15] so that we do git clone --mirror then we edit .git/config to aim for gerrit.wikimedia.org. [19:16:19] but needs an admin [19:16:35] since it needs forge commit and forge (something else) [19:16:54] paladox: well we already have the permanent mirror set up here: https://github.com/wikimedia/marvin [19:17:06] yep [19:17:07] oh [19:17:09] paladox: we just need to populate it with the commits from https://github.com/niedzielski/marvin [19:17:22] yeh, that one requires some one with the forge rights [19:17:29] paladox: i could try pushing to master on github but i didn't want to break the mirror (i'm not sure if it mirrors both ways) [19:17:39] uh [19:17:44] i will check [19:17:57] nope dosen't mirror to github [19:18:01] i mean gerrit [19:18:17] darn [19:18:20] you need to be added to https://gerrit.wikimedia.org/r/#/admin/groups/1348,members [19:18:28] so that you can grant your self the rights [19:18:29] :) [19:18:35] i tried pushing directly to the gerrit master branch but got a permission denied error [19:24:12] paladox: do you know who i would ask to get on that permisson list? [19:35:31] niedzielski you need to ask an gerrit admin (iirc most releng team members are admin) [19:41:16] Zppix: ok will do. thanks! [19:44:23] 10Release-Engineering-Team, 10Marvin: Merge github repo into gerrit repo - https://phabricator.wikimedia.org/T172702#3507714 (10Niedzielski) [19:59:06] niedzielski i belive releng :) [19:59:56] paladox: thanks! i've just add the #release-engineering-team tag to the ticket and tried to note the urls [20:00:05] ok thanks [20:23:32] 10Browser-Tests-Infrastructure, 10Release-Engineering-Team (Next), 10MinervaNeue, 10Readers-Web-Backlog, and 5 others: [4 hrs] MinervaNeue browser test are flaking (waiting for {:class=>"mw-notification", :tag_name=>"div"} to become present ) - https://phabricator.wikimedia.org/T170890#3446357 (10bmansurov... [20:27:07] 10Browser-Tests-Infrastructure, 10Release-Engineering-Team (Next), 10MinervaNeue, 10Readers-Web-Backlog, and 5 others: [4 hrs] MinervaNeue browser test are flaking (waiting for {:class=>"mw-notification", :tag_name=>"div"} to become present ) - https://phabricator.wikimedia.org/T170890#3507995 (10Jdlrobson... [20:41:10] 10Deployment-Systems, 10MediaWiki-extensions-CentralAuth: $wgLocalVirtualHosts should include login.wikimedia.org, wikidata.org and others? - https://phabricator.wikimedia.org/T172357#3508097 (10aaron) We should see if there is any unrelated downside to adding login/wikidata.org. I'd imagine it would be useful... [20:52:06] niedzielski i fixed the project tags. [21:04:43] Project selenium-MinervaNeue-BugFix » chrome,beta,Linux,BrowserTests build #2: 04FAILURE in 26 min: https://integration.wikimedia.org/ci/job/selenium-MinervaNeue-BugFix/BROWSER=chrome,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=BrowserTests/2/ [21:08:01] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban), 10Performance-Team, 10WebPageTest: Where to trigger WebPageTest jobs? - https://phabricator.wikimedia.org/T166756#3508173 (10Krinkle) [21:12:55] Project selenium-MinervaNeue-BugFix » firefox,beta,Linux,BrowserTests build #2: 04FAILURE in 35 min: https://integration.wikimedia.org/ci/job/selenium-MinervaNeue-BugFix/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=BrowserTests/2/ [21:18:52] 10Release-Engineering-Team, 10JobRunner-Service, 10Performance-Team, 10Regression: Investigate 30x increase in Jobrunner errors - https://phabricator.wikimedia.org/T171371#3508192 (10Krinkle) a:03aaron >>! In T171371#3481333, @gerritbot wrote: > Change 368116 merged by jenkins-bot: > [mediawiki/extension... [21:26:51] Zppix: thanks :] [21:26:59] np [21:27:10] oh geez i keep forgeting everyone is at wikimania D: [21:30:05] thats this week/ [21:30:06] ? [21:31:16] i think so [21:31:25] https://meta.wikimedia.org/wiki/Wikimania_2017 [22:08:41] 10Deployment-Systems, 10Release-Engineering-Team (Kanban): Automate the recurring management of wikitech:Deployments and phab:#train_deployments - https://phabricator.wikimedia.org/T114488#3508287 (10greg) See also: {T171940} [22:09:04] 10Deployment-Systems, 10wikitech.wikimedia.org, 10User-MarcoAurelio: Create a Gadget to easily add/remove/modify patches for SWAT at wikitech:Deployments - https://phabricator.wikimedia.org/T171940#3480850 (10greg) [23:12:08] (03PS2) 10Thcipriani: Assert MediaWiki does not generate error logs [integration/config] - 10https://gerrit.wikimedia.org/r/369676 (https://phabricator.wikimedia.org/T50002) (owner: 10Hashar) [23:13:51] (03CR) 10Thcipriani: Assert MediaWiki does not generate error logs (032 comments) [integration/config] - 10https://gerrit.wikimedia.org/r/369676 (https://phabricator.wikimedia.org/T50002) (owner: 10Hashar)