[00:01:13] 10Release-Engineering-Team (Kanban), 10Phabricator (2017-06-14), 10Upstream: Clicking "Add Existing Panel" and entering `Wxxx` in the field shows no results - https://phabricator.wikimedia.org/T166236#3289400 (10mmodell) [00:04:18] 10Release-Engineering-Team (Kanban), 10Phabricator (2017-06-14), 10Upstream: Clicking "Add Existing Panel" and entering `Wxxx` in the field shows no results - https://phabricator.wikimedia.org/T166236#3350336 (10mmodell) 05Open>03Resolved [00:05:15] 10Release-Engineering-Team (Kanban), 10Release, 10Train Deployments: MW-1.30.0-wmf.6 deployment blockers - https://phabricator.wikimedia.org/T167535#3350339 (10greg) (I added this task to that patch to have gerritbot comment when it merges, but I did it when Phab was momentarily down for routine maintenance... [00:05:46] twentyafterfour: this is annoying, bad gerritbot https://phabricator.wikimedia.org/T167535#3350339 [00:06:22] :( [00:07:02] yeah, a dumb bot just needs to be made smarter, but... whatever [00:07:06] just noting :) [00:07:21] greg-g: there is a better way to implement what gerritbot does [00:07:42] since commits can be associated with tasks we just need to use that feature to track when commits merge [00:08:14] e.g. edit related objects -> add the hash of the commit -> some new functionality tracks when that same commit appears on a tracked branch (rather than refs/changes/*) [00:08:24] * greg-g nods [00:09:22] I've been wanting to do that for a while but we're kinda stuck in a situation where investment in gerrit seems iffy and investment in differential is stalled [00:10:34] 10Release-Engineering-Team (Kanban), 10Release, 10Train Deployments: MW-1.30.0-wmf.6 deployment blockers - https://phabricator.wikimedia.org/T167535#3350341 (10mmodell) [00:11:02] twentyafterfour: yeah, awesome huh? [00:11:15] yep :) [00:11:56] Well I don't know how hard it would be to implement but essentially we just need to look at the "branches" field on https://phabricator.wikimedia.org/rMW94749fa62e95407fc82d9875caf5434a97d4620f [00:12:15] should be possible to query that from within https://phabricator.wikimedia.org/T167535 [00:12:35] in the details section, the commit could have a different icon to indicate merged vs. pending in gerrit [00:13:06] oh so task types are now a thing [00:13:17] https://phabricator.wikimedia.org/maniphest/task/edit/form/34/ [00:13:37] which creates release tasks like this: [00:13:39] https://phabricator.wikimedia.org/T167893 [00:15:12] neato! [00:24:24] 10Release-Engineering-Team (Kanban), 10Release, 10Train Deployments: MW-1.30.0-wmf.9 deployment blockers - https://phabricator.wikimedia.org/T167893#3350396 (10greg) [00:34:59] So, do any of the still-awake releng folks know how to fix beta-scap-eqiad? [00:35:01] ( twentyafterfour greg-g ) [00:35:14] See also my discussion with RainbowSprinkles and Reedy an hour ago [00:39:56] looks like https://www.mediawiki.org/wiki/Continuous_integration/Jenkins#Hung_beta_code.2Fdb_update ? [00:40:20] I should be able to fix it. It looks like the jenkins slave agent may have gone wonky on tin [00:40:43] cool, thanks man, I was just about to walk away :/ [00:40:54] np :) [00:52:21] !log deployment-tin jenkins agent borked for 4 hours, should be fixed now [00:52:26] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [00:52:45] got a beta-scap-eqiad running at least https://integration.wikimedia.org/ci/job/beta-scap-eqiad/159784/console [01:08:02] success \o/ [01:34:33] PROBLEM - Puppet staleness on deployment-aqs01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [43200.0] [01:37:38] 10Release-Engineering-Team (Kanban), 10Operations, 10Patch-For-Review, 10Security-General: setup releases1001.eqiad.wmnet (was: setup mwreleases1001) - https://phabricator.wikimedia.org/T164030#3350488 (10Dzahn) reinstalled as releases1001, with stretch. the "releasers-mediawiki" group has shell (again). w... [04:17:07] Yippee, build fixed! [04:17:08] Project selenium-MultimediaViewer » firefox,beta,Linux,BrowserTests build #423: 09FIXED in 21 min: https://integration.wikimedia.org/ci/job/selenium-MultimediaViewer/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=BrowserTests/423/ [05:55:21] Project beta-scap-eqiad build #159815: 04FAILURE in 1 min 40 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/159815/ [06:05:48] Yippee, build fixed! [06:05:49] Project beta-scap-eqiad build #159816: 09FIXED in 2 min 6 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/159816/ [06:22:30] Yippee, build fixed! [06:22:31] Project selenium-Wikibase » chrome,test,Linux,BrowserTests build #392: 09FIXED in 1 hr 42 min: https://integration.wikimedia.org/ci/job/selenium-Wikibase/BROWSER=chrome,MEDIAWIKI_ENVIRONMENT=test,PLATFORM=Linux,label=BrowserTests/392/ [06:44:43] Yippee, build fixed! [06:44:43] Project selenium-Wikibase » chrome,beta,Linux,BrowserTests build #392: 09FIXED in 2 hr 4 min: https://integration.wikimedia.org/ci/job/selenium-Wikibase/BROWSER=chrome,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=BrowserTests/392/ [07:23:32] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban), 10Patch-For-Review, 10Zuul: Zuul refused to start from contint1001 - https://phabricator.wikimedia.org/T167833#3350627 (10hashar) Ah thank you Tyler. I guess the definitive solution is to migrate to a proper systemd service defi... [07:25:05] 10Release-Engineering-Team (Kanban), 10MinervaNeue, 10Reading-Web-Backlog, 10Patch-For-Review: Skins cannot run browser tests per commit - https://phabricator.wikimedia.org/T167543#3350628 (10hashar) \O/ [07:26:24] 10MediaWiki-Codesniffer, 10Patch-For-Review: No sniff for "function (" versus "function(" - https://phabricator.wikimedia.org/T149623#3350629 (10hashar) That has been implemented now. Guess the task will be solved when the next code sniffer version is cut? [08:12:04] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban), 10Performance-Team, 10Jenkins, 10Upstream: WebPageTest job fails intemittently with "java.io.IOException: Unexpected termination of the channel" - https://phabricator.wikimedia.org/T166557#3350676 (10hashar) https://integration... [08:24:20] hashar: hey, good morning. https://gerrit.wikimedia.org/r/354522 Is blocking some of my work on wikibase. Can you take a look when you have some free time? [08:32:33] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban), 10Performance-Team, 10Jenkins, 10Upstream: WebPageTest job fails intemittently with "java.io.IOException: Unexpected termination of the channel" - https://phabricator.wikimedia.org/T166557#3350692 (10hashar) I found a bunch of... [08:59:53] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban), 10Performance-Team, 10Jenkins, 10Upstream: WebPageTest job fails intemittently with "java.io.IOException: Unexpected termination of the channel" - https://phabricator.wikimedia.org/T166557#3350744 (10hashar) All the stacktraces... [09:09:58] 10Release-Engineering-Team (Kanban), 10Wikidata, 10Story: [Story] Use composer-merge-plugin to include Wikidata components in mediawiki-vendor - https://phabricator.wikimedia.org/T95663#3350783 (10Ladsgroup) [09:10:03] 10Gerrit, 10Wikidata: [Task] move git repositories that are dependencies of wikidata to gerrit - https://phabricator.wikimedia.org/T74907#3350784 (10Ladsgroup) [09:21:05] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban), 10Performance-Team, 10Jenkins, 10Upstream: WebPageTest job fails intemittently with "java.io.IOException: Unexpected termination of the channel" - https://phabricator.wikimedia.org/T166557#3350831 (10hashar) From a `zgrep -c 'I... [10:22:26] PROBLEM - Long lived cherry-picks on puppetmaster on deployment-puppetmaster02 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [10:39:01] !log apt-get upgrade on deployment-tin [10:39:04] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [10:49:53] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban), 10Performance-Team, 10Jenkins, 10Upstream: WebPageTest job fails intemittently with "java.io.IOException: Unexpected termination of the channel" - https://phabricator.wikimedia.org/T166557#3351007 (10hashar) I am absolutely at... [10:55:24] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban), 10Performance-Team, 10Jenkins, 10Upstream: WebPageTest job fails intermittently with "java.io.IOException: Unexpected termination of the channel" - https://phabricator.wikimedia.org/T166557#3351024 (10Aklapper) [10:59:31] PROBLEM - Puppet errors on deployment-pdf01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [11:16:17] PROBLEM - Puppet errors on deployment-pdfrender02 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [11:56:19] RECOVERY - Puppet errors on deployment-pdfrender02 is OK: OK: Less than 1.00% above the threshold [0.0] [12:23:42] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban), 10Performance-Team, 10Jenkins, 10Upstream: WebPageTest job fails intermittently with "java.io.IOException: Unexpected termination of the channel" - https://phabricator.wikimedia.org/T166557#3300065 (10Paladox) See https://issue... [13:04:47] PROBLEM - Puppet errors on deployment-conf03 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [13:09:44] 10Release-Engineering-Team, 10Operations, 10Wikimedia-log-errors: Cronjobs attempting to connect to labstestweb2001 - https://phabricator.wikimedia.org/T167961#3351232 (10Marostegui) [13:10:38] 10Release-Engineering-Team, 10Operations, 10Wikimedia-log-errors: Cronjobs attempting to connect to labstestweb2001 - https://phabricator.wikimedia.org/T167961#3351245 (10Marostegui) p:05Triage>03Normal [13:11:37] 10Release-Engineering-Team, 10Operations, 10cloud-services-team, 10Wikimedia-log-errors: Cronjobs attempting to connect to labstestweb2001 - https://phabricator.wikimedia.org/T167961#3351232 (10Marostegui) [13:25:51] 10Release-Engineering-Team, 10Operations, 10cloud-services-team, 10Wikimedia-log-errors: Cronjobs attempting to connect to labstestweb2001 - https://phabricator.wikimedia.org/T167961#3351232 (10Andrew) In general we're trying to make wikitech (and labtestwikitech) more like normal wikis... they're currentl... [13:43:23] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban), 10Operations, 10Patch-For-Review, 10Zuul: Migrate zuul-server behind systemd service - https://phabricator.wikimedia.org/T167845#3351369 (10hashar) a:03Paladox @Paladox is kindly dealing with it \O/ [13:47:04] Yippee, build fixed! [13:47:05] Project selenium-VisualEditor » firefox,beta,Linux,BrowserTests build #429: 09FIXED in 3 min 3 sec: https://integration.wikimedia.org/ci/job/selenium-VisualEditor/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=BrowserTests/429/ [14:17:53] Project beta-update-databases-eqiad build #17829: 15ABORTED in 45 min: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/17829/ [14:22:35] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban), 10Performance-Team, 10Jenkins, 10Upstream: WebPageTest job fails intermittently with "java.io.IOException: Unexpected termination of the channel" - https://phabricator.wikimedia.org/T166557#3351445 (10hashar) [14:22:57] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban), 10Operations, 10Patch-For-Review, 10Zuul: Migrate zuul-server behind systemd service - https://phabricator.wikimedia.org/T167845#3351446 (10Paladox) Thanks :) [14:33:44] Yippee, build fixed! [14:33:45] Project selenium-WikiLove » firefox,beta,Linux,BrowserTests build #424: 09FIXED in 1 min 43 sec: https://integration.wikimedia.org/ci/job/selenium-WikiLove/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=BrowserTests/424/ [14:36:00] 10Continuous-Integration-Infrastructure, 10Jenkins: mw-ext-php70-phan-jessie complains about PHP temp directory not writable to composer - https://phabricator.wikimedia.org/T167969#3351474 (10pmiazga) [14:39:28] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban), 10Performance-Team, 10Jenkins, 10Upstream: WebPageTest job fails intermittently with "java.io.IOException: Unexpected termination of the channel" - https://phabricator.wikimedia.org/T166557#3351497 (10hashar) [14:46:24] 10Scap, 10Discovery, 10Interactive-Sprint, 10Maps (Kartotherian), 10Patch-For-Review: Break Kartotherian scap3 deployment into 2 groups - https://phabricator.wikimedia.org/T147337#3351576 (10Gehel) This seems trivial enough, patch is written and waiting for review. [14:47:08] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban), 10Performance-Team, 10Jenkins, 10Upstream: WebPageTest job fails intermittently with "java.io.IOException: Unexpected termination of the channel" - https://phabricator.wikimedia.org/T166557#3351583 (10hashar) [14:58:27] 10Scap, 10Discovery, 10Interactive-Sprint, 10Maps (Kartotherian), 10Patch-For-Review: Break Kartotherian scap3 deployment into 2 groups - https://phabricator.wikimedia.org/T147337#3351646 (10Gehel) a:03Gehel [15:05:01] Project beta-update-databases-eqiad build #17830: 15ABORTED in 45 min: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/17830/ [15:22:45] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban), 10Performance-Team, 10Jenkins, 10Upstream: WebPageTest job fails intermittently with "java.io.IOException: Unexpected termination of the channel" - https://phabricator.wikimedia.org/T166557#3351717 (10Paladox) See https://wiki.... [15:31:22] 10Scap, 10Discovery, 10Interactive-Sprint, 10Maps (Kartotherian), 10Patch-For-Review: Break Kartotherian scap3 deployment into 2 groups - https://phabricator.wikimedia.org/T147337#3351735 (10debt) Cool! :) [15:47:35] Project selenium-MobileFrontend » chrome,beta,Linux,BrowserTests build #455: 04FAILURE in 25 min: https://integration.wikimedia.org/ci/job/selenium-MobileFrontend/BROWSER=chrome,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=BrowserTests/455/ [15:56:27] Yippee, build fixed! [15:56:27] Project selenium-MobileFrontend » firefox,beta,Linux,BrowserTests build #455: 09FIXED in 34 min: https://integration.wikimedia.org/ci/job/selenium-MobileFrontend/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=BrowserTests/455/ [15:57:01] thcipriani: hey hey. got a late start today and apparently i'm out of coffee! [15:57:16] can we push our meeting back by 10 min? [15:57:28] marxarelli: yeah, np [15:57:35] same hangout time same hangout place [15:57:41] well [15:57:45] different hangout time [15:57:48] :) [15:57:51] dope :) [15:58:14] luckily there is a bakery with coffee two blocks from my place [15:58:53] caffeine AND sugar [16:05:01] Project beta-update-databases-eqiad build #17831: 15ABORTED in 45 min: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/17831/ [16:05:48] uh oh, those haven't been completing... [16:07:49] Again? [16:07:51] grrrr [16:08:08] Time out [16:08:09] Hmm [16:08:25] ukwiki? [16:08:37] * RainbowSprinkles looks [16:12:43] 10Beta-Cluster-Infrastructure, 10Collaboration-Team-Triage, 10Flow: beta-update-databases-eqiad times out at "index flow_workflow_update_timestamp already" for ukwiki - https://phabricator.wikimedia.org/T167981#3351912 (10greg) [16:13:21] 10Release-Engineering-Team (Kanban), 10MW-1.30-release-notes, 10MediaWiki-Database, 10MediaWiki-Unit-tests, and 6 others: 1.28-alpha / Error: 42P01 ERROR: table "unittest_user_groups" does not exist - https://phabricator.wikimedia.org/T149454#3351938 (10demon) 05Open>03Resolved a:03demon [16:16:32] greg-g: Ahhh, wikidatawiki has a maintenance script it's running as part of update.php [16:16:40] huh [16:16:52] Yeah, it's got some unlogged update job [16:16:58] Needs to run & log at least once [16:17:31] Ahhhh, yeah. This gonna take awhile [16:17:42] 500k pages in batches of 1000! :) [16:18:00] .... [16:18:03] I imagine in production this runs async with the deployment [16:18:15] But for update.php workflows, it'll break the jenkins 45m timeout [16:18:25] Such is life sometimes :) [16:19:36] run it outside of jenkins? [16:22:21] I'm running it locally on deployment-tin right now [16:22:31] It's gonna take awhile to complete. The jenkins job will flap until then [16:22:36] Actually, lemme disable it [16:25:54] * greg-g nods [16:34:13] !log deployment-prep: Disabled database updates for awhile, running it by hand [16:34:17] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [16:50:01] Project beta-update-databases-eqiad build #17832: 15ABORTED in 45 min: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/17832/ [16:52:15] Oh, disabling didn't work? [16:52:18] * RainbowSprinkles fumes [16:52:37] I guess triggering it doesn't respect that [17:02:37] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team (Kanban), 10Collaboration-Team-Triage, 10Flow: beta-update-databases-eqiad times out at "index flow_workflow_update_timestamp already" for ukwiki - https://phabricator.wikimedia.org/T167981#3352107 (10greg) a:03demon ```lang=irc 16:34 <+RainbowSp>... [17:05:04] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team (Kanban), 10Collaboration-Team-Triage, 10Flow: beta-update-databases-eqiad times out at "index flow_workflow_update_timestamp already" for ukwiki - https://phabricator.wikimedia.org/T167981#3352113 (10demon) That's not actually where it's timing out... [17:05:13] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team (Kanban), 10Collaboration-Team-Triage, 10Flow: beta-update-databases-eqiad times out at "index flow_workflow_update_timestamp already" for ukwiki - https://phabricator.wikimedia.org/T167981#3352114 (10demon) p:05High>03Unbreak! [17:05:27] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team (Kanban), 10Collaboration-Team-Triage, 10Flow: beta-update-databases-eqiad times out on wikidatawiki (maintenance update) - https://phabricator.wikimedia.org/T167981#3352116 (10demon) [17:05:45] PROBLEM - Puppet errors on swift is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [17:07:19] PROBLEM - Puppet errors on swift-storage-01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [17:17:20] 10Browser-Tests-Infrastructure, 10Release-Engineering-Team (Backlog), 10CirrusSearch, 10Discovery: Support alternative API endpoints - https://phabricator.wikimedia.org/T99663#3352139 (10debt) This already works on CirrusSearch....removing #discovery-search tag. [17:43:42] PROBLEM - Puppet errors on deployment-etcd-01 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [18:02:04] 10Browser-Tests-Infrastructure: MobileFrontend Chrome browser test job has become unstable - https://phabricator.wikimedia.org/T167994#3352389 (10Jdlrobson) [18:02:12] 10Browser-Tests-Infrastructure, 10Release-Engineering-Team: MobileFrontend Chrome browser test job has become unstable - https://phabricator.wikimedia.org/T167994#3352401 (10Jdlrobson) p:05Triage>03High [18:04:05] 10Continuous-Integration-Infrastructure, 10Electron-PDFs, 10MediaWiki-Internationalization, 10Reading-Web-Backlog: Phan throws false positive when checking SpecialElectronPDF file - https://phabricator.wikimedia.org/T167995#3352413 (10pmiazga) [18:04:56] 10Continuous-Integration-Infrastructure, 10Electron-PDFs, 10MediaWiki-Internationalization, 10Reading-Web-Backlog: Phan throws false positive when checking SpecialElectronPDF file - https://phabricator.wikimedia.org/T167995#3352431 (10pmiazga) [18:05:01] Project beta-update-databases-eqiad build #17833: 15ABORTED in 45 min: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/17833/ [18:17:18] 10Continuous-Integration-Infrastructure, 10Electron-PDFs, 10MediaWiki-Internationalization, 10Reading-Web-Backlog: Phan throws false positive when checking SpecialElectronPDF file - https://phabricator.wikimedia.org/T167995#3352502 (10pmiazga) @Ladsgroup - any idea what can be wrong? [18:25:49] 10Release-Engineering-Team (Kanban), 10Release Pipeline (Blubber): Escape Blubber config values when compiling to Dockerfile - https://phabricator.wikimedia.org/T167999#3352546 (10dduvall) [18:31:26] 10Release-Engineering-Team (Kanban), 10Release Pipeline (Blubber): Escape Blubber config values when compiling to Dockerfile - https://phabricator.wikimedia.org/T167999#3352602 (10dduvall) p:05Triage>03Normal [18:34:17] 10Release-Engineering-Team (Kanban), 10Release Pipeline (Blubber): Improve Blubber unit test coverage - https://phabricator.wikimedia.org/T168001#3352610 (10dduvall) a:03dduvall [18:36:05] Yippee, build fixed! [18:36:06] Project selenium-MobileFrontend » chrome,beta,Linux,BrowserTests build #456: 09FIXED in 35 min: https://integration.wikimedia.org/ci/job/selenium-MobileFrontend/BROWSER=chrome,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=BrowserTests/456/ [18:45:20] when you login into Phab and have 2fa enable, and you do the first step but not the second, then you are half-logged in [18:45:35] it told me the number of notifications, but not the content of them [18:46:02] twentyafterfour ^^ that's a security problem right? [18:46:16] i dont think it can be that serious, it's just the number itself [18:46:20] but i still noticed [18:46:37] not security, people have notifications for any number of reasons [18:46:42] but yeah, bad state to be in [18:48:04] it may have been my fault. so i just cleaned all my cookies in my browser [18:48:08] i think thats a intented feature if you look at login history in settings it shows full login and partial login (even for me which idont use 2fa) [18:48:18] i didn't do a proper logout [18:48:39] aha, ok [18:50:11] (03CR) 10Bmansurov: [C: 031] Remove Cards from CI [integration/config] - 10https://gerrit.wikimedia.org/r/359044 (https://phabricator.wikimedia.org/T167452) (owner: 10Jdlrobson) [18:53:43] RECOVERY - Puppet errors on deployment-etcd-01 is OK: OK: Less than 1.00% above the threshold [0.0] [18:55:58] Project beta-scap-eqiad build #159891: 04FAILURE in 2 min 0 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/159891/ [19:05:01] Project beta-update-databases-eqiad build #17834: 15ABORTED in 45 min: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/17834/ [19:06:24] Yippee, build fixed! [19:06:25] Project beta-scap-eqiad build #159892: 09FIXED in 2 min 30 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/159892/ [19:13:24] jenkins with Java 8.. will it blend ?:) [19:13:29] is about to find out [19:13:44] stretch VM with jenkins class [19:16:00] 10Beta-Cluster-Infrastructure, 10media-storage, 10Patch-For-Review: deployment-ms-be03.deployment-prep and deployment-ms-be04.deployment-prep have high load / system CPU - https://phabricator.wikimedia.org/T160990#3352795 (10hashar) I tried a few things with `nscd`, have dig in the code a bit but I could not... [19:16:56] 10Continuous-Integration-Infrastructure, 10Electron-PDFs, 10MediaWiki-Internationalization, 10Reading-Web-Backlog: Phan throws false positive when checking SpecialElectronPDF file - https://phabricator.wikimedia.org/T167995#3352798 (10Ladsgroup) @pmiazga hey, the patch seems to be merged now. Is the issue... [19:19:24] 10Beta-Cluster-Infrastructure, 10media-storage, 10Patch-For-Review: deployment-ms-be03.deployment-prep and deployment-ms-be04.deployment-prep have high load / system CPU - https://phabricator.wikimedia.org/T160990#3352808 (10hashar) [19:19:42] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team (Kanban), 10media-storage, 10Patch-For-Review: deployment-ms-be03.deployment-prep and deployment-ms-be04.deployment-prep have high load / system CPU - https://phabricator.wikimedia.org/T160990#3117645 (10hashar) [19:20:44] 10Continuous-Integration-Infrastructure, 10Electron-PDFs, 10MediaWiki-Internationalization, 10Reading-Web-Backlog: Phan throws false positive when checking SpecialElectronPDF file - https://phabricator.wikimedia.org/T167995#3352813 (10pmiazga) Patch rMW3e28246796946e9673b0142d37130dced5635e86 causes Phan t... [19:25:50] Project beta-scap-eqiad build #159894: 04FAILURE in 1 min 54 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/159894/ [19:27:12] 10Browser-Tests-Infrastructure, 10Release-Engineering-Team: MobileFrontend Chrome browser test job has become unstable - https://phabricator.wikimedia.org/T167994#3352389 (10pmiazga) @Jdlrobson - lots of errors fail with `The Sauce VMs failed to start the browser or device. For more info, please check https://... [19:31:57] PROBLEM - Puppet errors on deployment-imagescaler01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [19:34:24] Yippee, build fixed! [19:34:24] Project beta-scap-eqiad build #159895: 09FIXED in 2 min 21 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/159895/ [19:44:52] Hi releng! [19:45:10] CiviCRM tests seem not to be triggered by gerrit patches now [19:45:24] e.g. https://gerrit.wikimedia.org/r/352746 [19:45:32] any ideas why? [19:46:51] jjb/wm-fundraising.yaml hasn't changed lately [19:46:55] ejegg: jenkins is in a bad mood? [19:46:58] jk, no idea :) [19:47:47] ejegg recheck seemed to fix it :) [19:49:04] weird, it didn't a minute ago on https://gerrit.wikimedia.org/r/359233 [19:49:22] trying again on that one [19:50:35] I think sometimes it misses rebases. [19:50:38] I've seen that before [20:05:01] Project beta-update-databases-eqiad build #17835: 15ABORTED in 45 min: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/17835/ [20:21:39] RainbowSprinkles hi, sorry to bother again, how do i add users to https://gerrit.wikimedia.org/r/#/admin/groups/1332,members ? [20:21:50] it shows blank out for me but im in that group [20:21:51] please [20:22:48] I said I'm busy. [20:26:09] 10Gerrit: Internal server error when accessing change - https://phabricator.wikimedia.org/T168012#3352953 (10cicalese) [20:28:10] 10Gerrit: Internal server error when accessing change - https://phabricator.wikimedia.org/T168012#3352953 (10Paladox) Hi, was this your change? If not was this a draft? If it was someone may have either deleted it or removed you as a reviewer. [20:30:58] 10Gerrit: Internal server error when accessing change - https://phabricator.wikimedia.org/T168012#3352967 (10cicalese) It was my change. I created it through the Gerrit user interface and used it to add a simple text file to my extension. After I tried to publish the edit, the change became inaccessible. Nobody... [20:37:23] 10Gerrit: Internal server error when accessing change - https://phabricator.wikimedia.org/T168012#3352974 (10Paladox) Himm, i get a similar error too, with my change https://gerrit.wikimedia.org/r/#/c/356181/ [20:40:08] 10Gerrit: Internal server error when accessing change - https://phabricator.wikimedia.org/T168012#3352976 (10cicalese) Interesting - I can access yours with no problem. [20:41:02] 10Gerrit: Internal server error when accessing change - https://phabricator.wikimedia.org/T168012#3352977 (10Paladox) Really? It's meant to be a draft. So i guess there's a bug. (lucky drafts are being replaced with something stable) [20:41:50] hmm strange, it throws 500 for me [20:41:58] but others it works. [20:42:08] is this about the new "task types" ? [20:42:27] nevermind [20:42:33] Nope [20:44:35] tasks types? [20:47:24] sorry, totally unrelated, but in phab there are new task types [20:48:01] O_o [20:50:16] 10Gerrit: Internal server error when accessing change - https://phabricator.wikimedia.org/T168012#3352996 (10cicalese) That's so strange! Yours is not listed as a draft when I look at it, but mine is marked as a draft in the list of changes. Should I just give up on this change and start over? Is there a way to... [20:51:56] 10Gerrit: Internal server error when accessing change - https://phabricator.wikimedia.org/T168012#3352998 (10Paladox) Uh, maybe reindexing the change will fix this. (but needs to be done either through ssh using online index or on the server (you need access to do both which only the admin currently has) [20:52:17] RainbowSprinkles: Any idea why https://gerrit.wikimedia.org/r/#/c/359191/ gives HTTP 500 ? [20:52:27] Its' one of these 4 - https://gerrit.wikimedia.org/r/#/q/topic:rl-bench+(status:open+OR+status:merged) [20:52:31] Krinkle strange [20:52:32] "Add benchmark for JSMinPlus" [20:52:38] works for me [20:52:48] we were talking about this on https://phabricator.wikimedia.org/T168012 [20:53:53] 10Gerrit, 10Release-Engineering-Team (Kanban), 10Upstream: "git review -d XXX" doesn't work for http gerrit - https://phabricator.wikimedia.org/T100987#3353003 (10kaldari) > The workaround are: > - use the ssh:// protocol for review > - fetch patches manually (eg: git fetch origin refs/changes/DE/ABCDE &... [20:54:32] 10Gerrit: Internal server error when accessing change - https://phabricator.wikimedia.org/T168012#3353004 (10Paladox) p:05Triage>03High Three users have reported this. One of them on irc having same problem, inaccessible for them but accessible by everyone else. Triaging. [20:55:28] (03CR) 10Chad: [C: 032] Remove Cards from CI [integration/config] - 10https://gerrit.wikimedia.org/r/359044 (https://phabricator.wikimedia.org/T167452) (owner: 10Jdlrobson) [21:00:31] (03Merged) 10jenkins-bot: Remove Cards from CI [integration/config] - 10https://gerrit.wikimedia.org/r/359044 (https://phabricator.wikimedia.org/T167452) (owner: 10Jdlrobson) [21:05:01] Project beta-update-databases-eqiad build #17836: 15ABORTED in 45 min: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/17836/ [21:08:44] 10Gerrit: Internal server error when accessing change - https://phabricator.wikimedia.org/T168012#3353025 (10Paladox) ah https://gerrit.wikimedia.org/r/#/c/356181/ that's not a draft whoops, thought it was since i have several similar with same name but are a draft. But i get 500 when visiting it logged in. [21:13:44] 10Continuous-Integration-Config, 10VisualEditor: Intermittent failures of mwext-qunit-jessie - https://phabricator.wikimedia.org/T163123#3353034 (10Jdlrobson) Not seen this. Is it still happening? [21:13:46] 10Gerrit: Internal server error when accessing change - https://phabricator.wikimedia.org/T168012#3353036 (10Aklapper) Being logged in and going to https://gerrit.wikimedia.org/r/359241 I get: > Code Review - Error > The page you requested was not found, or you do not have permission to view this page. No "Inter... [21:15:04] 10Gerrit: Internal server error when accessing change - https://phabricator.wikimedia.org/T168012#3353041 (10Paladox) >>! In T168012#3353036, @Aklapper wrote: > Being logged in and going to https://gerrit.wikimedia.org/r/359241 I get: >> Code Review - Error >> The page you requested was not found, or you do not... [21:17:09] 10Gerrit: Internal server error when accessing change - https://phabricator.wikimedia.org/T168012#3353052 (10Paladox) Strange within the last minute 500 has shown when logged out for my change now. [21:17:24] 10Gerrit: Internal server error when accessing change - https://phabricator.wikimedia.org/T168012#3353055 (10Paladox) {F8464865} [21:28:22] haha [21:28:23] aha [21:28:29] mutante found the error [21:28:40] and it describes the error i got upstream [21:28:46] which is fixed in gerrit 2.14+ [21:30:41] paladox: ah :) [21:30:56] Yep [21:30:57] * paladox finds change [21:31:07] it also explains why it works for others [21:32:57] what is different for you? [21:33:04] not browser? [21:33:21] https://bugs.chromium.org/p/gerrit/issues/detail?id=6176 [21:33:24] mutante ^^ [21:34:14] 10Gerrit: Internal server error when accessing change - https://phabricator.wikimedia.org/T168012#3353152 (10Paladox) @Dzahn kindly looked in the logs and found this error for my change [2017-06-15 21:25:28,800] [HTTP-67550] ERROR com.google.gerrit.httpd.restapi.RestApiServlet : Error in GET /r/changes/356181/r... [21:35:11] so it's because of polygerrit? [21:35:34] grmbls:) [21:35:35] Nope [21:35:40] gwtui was affected [21:36:02] i didnt know the same error was causing gwtui problems at the time [21:36:20] it's to do with the new update they did to allow us to store reviewed flags in other db's. [21:36:27] i still dont understand what made you different from the other users [21:36:35] chrome? [21:36:50] mutante all browsers are affected. [21:36:55] per the error you got [21:37:12] and it is actually same error described in https://bugs.chromium.org/p/gerrit/issues/detail?id=6176 [21:37:14] you just told me how it works for other users (and it does work for me) [21:37:16] yes [21:37:23] mutante because i have the reviewed flag [21:37:24] then how can it be "all are affected" at the same time [21:37:30] and what is the difference [21:37:34] aha [21:37:42] mutante it explains alot [21:37:51] users haven't reviewed the files [21:37:59] so they doint have any reviewed flag on the file [21:38:05] aha [21:38:14] now at least that is a real difference, yep :) [21:38:33] yep, somehow my reviewer flag got remove from the db [21:38:36] causing my problem [21:38:45] it carn't find it so it is throwing null. [21:41:02] so db needs a reindex? [21:41:06] 10Gerrit: Internal server error when accessing change - https://phabricator.wikimedia.org/T168012#3353181 (10Paladox) @cicalese your error sounds different to mine. Yours sounds like the change is deleted or somehow gerrit does not think your the owner of the change. [21:41:17] Zppix i doint think reindex will fix that [21:42:44] upstream had this problem when they introduced that new "improvements in “reviewed” flags cache" improvements [21:43:05] PROBLEM - jenkins_service_running on releases1001 is CRITICAL: PROCS CRITICAL: 0 processes with regex args ^/usr/bin/java .*-jar /usr/share/jenkins/jenkins.war [21:43:38] though it was only experienced on 2.14 not 2.13 until now. [21:44:17] https://gerrit-review.googlesource.com/#/c/103373/ [21:49:48] ahah [21:50:28] mutante changes created throw web ui are getting corrupted. I can reproduce by creating change on mediawiki/core and then trying to publish the edit. [21:52:01] then let's disable it? [21:52:16] mutante nah [21:52:25] you carn't, [21:52:55] anyways sometimes it works and other times it dosen't. [21:52:56] bummer [21:53:06] there's a fix at least [21:53:10] that will fix the 500 [21:53:13] good [21:54:54] mutante we can backport the change i think [21:56:34] 10Gerrit: Internal server error when accessing change - https://phabricator.wikimedia.org/T168012#3353250 (10Paladox) @cicalese ah, i can confirm his problem is the same as mine now. Creating changes through web ui and trying to publish with changes done to file is throwing 500 for me too. [21:58:26] ACKNOWLEDGEMENT - jenkins_service_running on releases1001 is CRITICAL: PROCS CRITICAL: 0 processes with regex args ^/usr/bin/java .*-jar /usr/share/jenkins/jenkins.war daniel_zahn new install - https://gerrit.wikimedia.org/r/#/c/359227/2 will fix [22:00:19] * paladox backported the fix here https://gerrit-review.googlesource.com/110410 [22:05:42] Project beta-update-databases-eqiad build #17837: 15ABORTED in 45 min: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/17837/ [22:10:43] 10Gerrit: Internal server error when accessing change - https://phabricator.wikimedia.org/T168012#3353298 (10Paladox) I filled this https://bugs.chromium.org/p/gerrit/issues/detail?id=6519 upstream. Upstream are not planning for any more 2.13 releases so to fix this. we can a: build from the 2.13 branch with the... [22:24:03] 10Gerrit, 10Release-Engineering-Team: Internal server error when accessing change - https://phabricator.wikimedia.org/T168012#3353341 (10Paladox) Adding releng project as this is a gerrit source code problem. [22:34:31] 10Gerrit, 10Release-Engineering-Team: Internal server error when accessing change - https://phabricator.wikimedia.org/T168012#3353380 (10cicalese) Thank you for investigating this, @Paladox! [22:34:49] 10Gerrit, 10Release-Engineering-Team: Internal server error when accessing change - https://phabricator.wikimedia.org/T168012#3353381 (10Paladox) Your welcome :). [22:43:16] 10Release-Engineering-Team (Kanban), 10Release, 10Train Deployments: MW-1.30.0-wmf.5 deployment blockers - https://phabricator.wikimedia.org/T167533#3353389 (10MacFan4000) 05Open>03Resolved wmf.5 successfully deployed. [23:05:01] Project beta-update-databases-eqiad build #17838: 15ABORTED in 45 min: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/17838/ [23:33:06] RECOVERY - jenkins_service_running on releases1001 is OK: PROCS OK: 1 process with regex args ^/usr/bin/java .*-jar /usr/share/jenkins/jenkins.war [23:33:52] ^ i copied the jessie package to stretch [23:47:18] PROBLEM - Puppet errors on deployment-pdfrender02 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [23:52:32] PROBLEM - Free space - all mounts on deployment-fluorine02 is CRITICAL: CRITICAL: deployment-prep.deployment-fluorine02.diskspace._srv.byte_percentfree (<33.33%) [23:57:37] 10Gerrit, 10Labs, 10wikitech.wikimedia.org: Request to rename LegoFan4000 to MacFan4000 on WikiTech - https://phabricator.wikimedia.org/T165624#3353564 (10bd808) ``` $ ldapmodify -v -D 'uid=novaadmin,ou=people,dc=wikimedia,dc=org' -W -f MacFan4000.ldif ldap_initialize( ) Enter LDAP Password: replac...