[02:16:57] 10Gerrit, 10Release-Engineering-Team (Kanban), 10Cleanup, 10utfnormal: Archive php-normal which is unused - https://phabricator.wikimedia.org/T174583#3578675 (10demon) 05Resolved>03Open Oh should be dropped from extension meta repo too [04:56:50] PROBLEM - Mediawiki Error Rate on graphite-labs is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [10.0] [05:36:50] PROBLEM - Mediawiki Error Rate on graphite-labs is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [10.0] [06:22:36] (03CR) 10Legoktm: "Maybe this is a silly question...but why can't we use a base image of Debian Stretch? That would have PHP 7.0 without needing a custom apt" [integration/config] - 10https://gerrit.wikimedia.org/r/371708 (owner: 10Addshore) [06:25:16] (03CR) 10Legoktm: "Yeah, the depends-on stuff is all handled by zuul & zuul-cloner. We should use that instead of re-inventing it." [integration/config] - 10https://gerrit.wikimedia.org/r/371708 (owner: 10Addshore) [06:31:30] (03CR) 10Legoktm: WIP Docker: contint-mediawiki-extensions-phan (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/371708 (owner: 10Addshore) [06:57:18] PROBLEM - Puppet errors on deployment-kafka01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [08:19:02] (03CR) 10Thiemo Mättig (WMDE): "Just recently the Wikidata team tried to combine efforts on the code style rules the team agrees on (and they are still very different fro" (031 comment) [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/375547 (https://phabricator.wikimedia.org/T167694) (owner: 10Legoktm) [08:39:27] 10Continuous-Integration-Infrastructure (phase-out-trusty), 10Release-Engineering-Team (Kanban), 10Patch-For-Review: Migrate PHP5.5 jobs from Trusty to Jessie - https://phabricator.wikimedia.org/T161882#3579187 (10hashar) [08:44:32] PROBLEM - English Wikipedia Mobile Main page on beta-cluster is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Service Unavailable - string 'Wikipedia' not found on 'https://en.m.wikipedia.beta.wmflabs.org:443/wiki/Main_Page?debug=true' - 1953 bytes in 0.023 second response time [08:49:36] RECOVERY - English Wikipedia Mobile Main page on beta-cluster is OK: HTTP OK: HTTP/1.1 200 OK - 39483 bytes in 3.598 second response time [08:56:49] RECOVERY - Mediawiki Error Rate on graphite-labs is OK: OK: Less than 1.00% above the threshold [1.0] [08:57:31] PROBLEM - Puppet errors on deployment-imagescaler02 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [09:17:29] (03CR) 10Addshore: "Is there a wikimedia base debian stretch image already?" [integration/config] - 10https://gerrit.wikimedia.org/r/371708 (owner: 10Addshore) [09:19:09] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Backlog), 10Patch-For-Review, 10User-Addshore, 10Zuul: Upgrade pbr for zuul - https://phabricator.wikimedia.org/T162787#3579221 (10Addshore) [09:35:21] 10Release-Engineering-Team, 10Zuul: Remove pbr from zuul - https://phabricator.wikimedia.org/T174965#3579250 (10Paladox) 05Open>03declined [09:57:35] 10Beta-Cluster-Infrastructure, 10Multimedia, 10Thumbor, 10Patch-For-Review: On beta commons, thumbnailing of 3D files is broken still - https://phabricator.wikimedia.org/T170444#3579329 (10Gilles) [09:58:23] PROBLEM - nodepoold running on labnodepool1001 is CRITICAL: PROCS CRITICAL: 0 processes with UID = 113 (nodepool), regex args ^/usr/bin/python /usr/bin/nodepoold -d [09:59:32] RECOVERY - nodepoold running on labnodepool1001 is OK: PROCS OK: 1 process with UID = 113 (nodepool), regex args ^/usr/bin/python /usr/bin/nodepoold -d [10:02:07] PROBLEM - Free space - all mounts on deployment-kafka01 is CRITICAL: CRITICAL: deployment-prep.deployment-kafka01.diskspace.root.byte_percentfree (<33.33%) [10:09:32] PROBLEM - nodepoold running on labnodepool1001 is CRITICAL: PROCS CRITICAL: 0 processes with UID = 113 (nodepool), regex args ^/usr/bin/python /usr/bin/nodepoold -d [10:10:40] PROBLEM - Puppet errors on saucelabs-02 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [10:26:42] RECOVERY - nodepoold running on labnodepool1001 is OK: PROCS OK: 1 process with UID = 113 (nodepool), regex args ^/usr/bin/python /usr/bin/nodepoold -d [10:32:33] RECOVERY - Puppet errors on deployment-imagescaler02 is OK: OK: Less than 1.00% above the threshold [0.0] [10:45:41] RECOVERY - Puppet errors on saucelabs-02 is OK: OK: Less than 1.00% above the threshold [0.0] [11:15:12] (03PS1) 10Hashar: Switch MediaWiki core/vendor php5.5 to Jessie [integration/config] - 10https://gerrit.wikimedia.org/r/375992 (https://phabricator.wikimedia.org/T161882) [11:52:13] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Watching / External), 10Cloud-VPS, 10Nodepool, and 2 others: figure out if nodepool is overwhelming rabbitmq and/or nova - https://phabricator.wikimedia.org/T170492#3579592 (10hashar) It happened again this morning after I entered a fault... [11:53:19] (03CR) 10Paladox: [C: 031] Switch MediaWiki core/vendor php5.5 to Jessie [integration/config] - 10https://gerrit.wikimedia.org/r/375992 (https://phabricator.wikimedia.org/T161882) (owner: 10Hashar) [11:56:07] 10Release-Engineering-Team, 10Wikidata: Decide what to do with Wikibase JS-only libraries regarding the build/deployment of Wikidata code - https://phabricator.wikimedia.org/T174922#3579637 (10Addshore) >>! In T174922#3578222, @demon wrote: > Option #3 seems easiest to me from my armchair. Submodules do //not/... [12:11:31] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Watching / External), 10Cloud-VPS, 10Nodepool, and 2 others: figure out if nodepool is overwhelming rabbitmq and/or nova - https://phabricator.wikimedia.org/T170492#3579682 (10hashar) Another thing I have noticed on the [[https://grafana.... [12:15:27] (03CR) 10Hashar: [C: 032] Switch MediaWiki core/vendor php5.5 to Jessie [integration/config] - 10https://gerrit.wikimedia.org/r/375992 (https://phabricator.wikimedia.org/T161882) (owner: 10Hashar) [12:16:23] (03Merged) 10jenkins-bot: Switch MediaWiki core/vendor php5.5 to Jessie [integration/config] - 10https://gerrit.wikimedia.org/r/375992 (https://phabricator.wikimedia.org/T161882) (owner: 10Hashar) [12:22:56] Yippee, build fixed! [12:22:57] Project selenium-GettingStarted » firefox,beta,Linux,BrowserTests build #516: 09FIXED in 55 sec: https://integration.wikimedia.org/ci/job/selenium-GettingStarted/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=BrowserTests/516/ [12:46:55] hashar i wonder when we get to gerrit 2.15, should we grant the jenkins bot private changes rights? [12:47:04] i belive it has drafts rights currently. [13:25:46] (03PS1) 10Hashar: Migrate phpunit-coverage-publish to Jessie [integration/config] - 10https://gerrit.wikimedia.org/r/376009 (https://phabricator.wikimedia.org/T161882) [13:28:36] (03PS1) 10Aude: Update Wikidata - wmf/1.30.0-wmf.17 [tools/release] - 10https://gerrit.wikimedia.org/r/376010 [13:28:51] (03CR) 10Aude: [C: 032] Update Wikidata - wmf/1.30.0-wmf.17 [tools/release] - 10https://gerrit.wikimedia.org/r/376010 (owner: 10Aude) [13:29:25] (03Merged) 10jenkins-bot: Update Wikidata - wmf/1.30.0-wmf.17 [tools/release] - 10https://gerrit.wikimedia.org/r/376010 (owner: 10Aude) [13:30:50] (03PS1) 10Hashar: remexhtml publish: use YAML aliasing (DRY) [integration/config] - 10https://gerrit.wikimedia.org/r/376011 [13:31:56] (03CR) 10Hashar: [C: 032] "Jobs updated." [integration/config] - 10https://gerrit.wikimedia.org/r/376009 (https://phabricator.wikimedia.org/T161882) (owner: 10Hashar) [13:34:13] (03Merged) 10jenkins-bot: Migrate phpunit-coverage-publish to Jessie [integration/config] - 10https://gerrit.wikimedia.org/r/376009 (https://phabricator.wikimedia.org/T161882) (owner: 10Hashar) [13:35:25] 10Release-Engineering-Team, 10Wikidata, 10Epic, 10User-Addshore: [Epic] Kill the Wikidata build step - https://phabricator.wikimedia.org/T173818#3579974 (10Addshore) [13:39:27] 10Release-Engineering-Team, 10Wikidata, 10Epic, 10User-Addshore: [Epic] Kill the Wikidata build step - https://phabricator.wikimedia.org/T173818#3580005 (10Addshore) [13:40:00] 10Gerrit, 10Wikidata, 10User-Ladsgroup, 10Wikidata-Sprint-2016-03-01, 10Wikidata-Sprint-2016-04-12: [Task] Move DataTypes repository from Github to gerrit - https://phabricator.wikimedia.org/T127292#3580009 (10Addshore) @Ladsgroup any movement here? [14:01:04] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team: Disable xdebug for phpunit/composer unless needed - https://phabricator.wikimedia.org/T175028#3580068 (10hashar) [14:05:05] (03PS1) 10Hashar: Delete mwext-Wikibase-repo-tests-sqlite-php55-trusty [integration/config] - 10https://gerrit.wikimedia.org/r/376018 (https://phabricator.wikimedia.org/T161882) [14:12:18] hi! adding private values to mw config is still a git commit to tin:/srv/mediawiki-config/private yes? context is T144479 [14:12:19] T144479: Ensure thumbor container access is preserved by mw filebackend setzoneaccess - https://phabricator.wikimedia.org/T144479 [14:16:55] 10Continuous-Integration-Infrastructure (phase-out-trusty), 10Release-Engineering-Team (Kanban): Package php modules for Zend 5.5 on Jessie - https://phabricator.wikimedia.org/T174972#3580122 (10hashar) [14:17:27] godog: yeah [14:18:56] neat, thanks Reedy /cc gilles [14:33:44] Yippee, build fixed! [14:33:44] Project selenium-WikiLove » firefox,beta,Linux,BrowserTests build #507: 09FIXED in 1 min 43 sec: https://integration.wikimedia.org/ci/job/selenium-WikiLove/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=BrowserTests/507/ [14:35:19] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Watching / External), 10Cloud-VPS, 10Nodepool, 10cloud-services-team (Kanban): rabbitmq: Consume and log messages sent to notifications.error - https://phabricator.wikimedia.org/T175029#3580173 (10Andrew) [15:05:57] (03PS1) 10Hashar: Phase out legacy operations/puppet jobs [integration/config] - 10https://gerrit.wikimedia.org/r/376034 [15:07:14] 10Release-Engineering-Team (Watching / External), 10Operations, 10hardware-requests: eqiad: replacement tin/deployment server - https://phabricator.wikimedia.org/T174452#3580285 (10RobH) a:03mark Assigning to @mark for approval of spare server usage. @Mark: We have 4 total spare systems on the shelf ident... [15:19:10] 10Beta-Cluster-Infrastructure, 10MediaWiki-Authentication-and-authorization, 10MediaWiki-extensions-CentralAuth, 10MW-1.30-release-notes (WMF-deploy-2017-08-08_(1.30.0-wmf.13)), 10Patch-For-Review: "Loss of session data" on Beta Cluster - https://phabricator.wikimedia.org/T172560#3580324 (10Anomie) Note... [15:52:41] 10Beta-Cluster-Infrastructure, 10MediaWiki-Authentication-and-authorization, 10MediaWiki-extensions-CentralAuth, 10MW-1.30-release-notes (WMF-deploy-2017-08-08_(1.30.0-wmf.13)), 10Patch-For-Review: "Loss of session data" on Beta Cluster - https://phabricator.wikimedia.org/T172560#3580461 (10jmatazzoni) @... [16:00:00] 10Beta-Cluster-Infrastructure, 10Analytics: deployment-kafka01 - disk is full - https://phabricator.wikimedia.org/T174742#3580490 (10elukey) I tried to add api_version to the kafka Hiera config in https://wikitech.wikimedia.org/wiki/Hiera:Deployment-prep and restart eventlogging but it didn't count much, still... [16:01:49] I've made a change to PrivateSettings.php, what's the process for deploying it? scheduling it in a SWAT? [16:02:16] it's committed on tin (and its equivalent on deployment-tin for that matter) [16:02:48] Probably [16:02:53] Or just sync-file it [16:04:17] I've never used scap, I guess it's a pretty simple first use case to try [16:04:53] also could be the smallest way to take down the site, I guess a learning experience either way right [16:05:02] Well, as long as it's valid php... [16:06:26] I'll do that later tonight when I don't have a child to look after at the same time [16:10:34] Hi! Anyone know whom should I ask for help fixing my 2-factor auth for Phab? thx in advance!! [16:10:48] (currently locked out because phone with authenticator died...) [16:11:13] well well well, isn't that exactly what someone from the NSA would say? ;) [16:11:33] gilles: I'm somewhat amused you've never had to scap anything yet :) [16:11:40] AndyRussG: Fixing what about it? [16:11:56] I have to admit that I've been freeloading on the fine SWAT folks for some time [16:18:23] 10Beta-Cluster-Infrastructure, 10Analytics: deployment-kafka01 - disk is full - https://phabricator.wikimedia.org/T174742#3580565 (10Ottomata) Hm, I think we should just make this go away quickly for now, and then re-do deployment-prep kafka instances as part of the Kafka 0.11 upgrade process. We'll do this b... [16:25:10] PROBLEM - Long lived cherry-picks on puppetmaster on deployment-puppetmaster02 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [16:25:11] 10Scap: scap sync failed on i18n - https://phabricator.wikimedia.org/T175041#3580615 (10Mattflaschen-WMF) [16:25:19] 10Scap: scap sync failed on i18n - https://phabricator.wikimedia.org/T175041#3580630 (10Mattflaschen-WMF) p:05Triage>03High [16:27:16] 10Scap: scap sync failed on i18n - https://phabricator.wikimedia.org/T175041#3580615 (10Mattflaschen-WMF) I Ctrl-C'ed when I saw the problem, reverted the i18n changes, then did a sync-dir of everything (in progress). I think I reverted early enough above it did not start deploying to actual machines. [16:32:13] Reedy: I use the Google Authenticator app on my phone ( + oAuth from Mediawiki) to log into Phab... Mmm my phone died... (I guess I should have had some backup plan...) [16:32:28] Are you still logged into phab? [16:32:33] Reedy: nope [16:32:49] 10Scap: scap sync failed on i18n - https://phabricator.wikimedia.org/T175041#3580661 (10Mattflaschen-WMF) >>! In T175041#3580630, @Mattflaschen-WMF wrote: > I Ctrl-C'ed when I saw the problem, reverted the i18n changes, then did a sync-dir of everything (in progress). I think I reverted early enough above it di... [16:32:56] I guess you want to catch andre or twentyafterfour [16:33:20] Reedy: K cool, thx! :) [16:33:57] AndyRussG: it's possible to strip the 2nd factor from your account but I need some way to verify that it's you [16:37:16] 10Scap: scap sync failed on i18n - https://phabricator.wikimedia.org/T175041#3580674 (10Mattflaschen-WMF) I'm not 100% sure, but I don't think it deployed to the canaries before either, since there is no "check-canaries:" line. [16:37:49] 10Scap: scap sync failed on i18n - https://phabricator.wikimedia.org/T175041#3580676 (10Mattflaschen-WMF) Cleanup sync-dir: {P5959} [16:38:39] 10Scap: scap sync failed on i18n - https://phabricator.wikimedia.org/T175041#3580678 (10Mattflaschen-WMF) [16:39:02] twentyafterfour: cool! Mmm maybe a quick hangout? Thx!!! [16:39:37] (I was able to update my Google authenticator stuff, since I hadn't logged out there before it was too late...) [16:41:04] 10Gerrit, 10Patch-For-Review, 10User-Ladsgroup: Make gerrit use the new WMF logo - https://phabricator.wikimedia.org/T174576#3566488 (10fgiunchedi) FWIW this is the svg rendering I got on chrome 60 on Debian stretch before the revert during puppet swat today {F9342590} [16:42:12] (03PS1) 10Florianschmidtwelzow: Archive extension AWS [integration/config] - 10https://gerrit.wikimedia.org/r/376054 (https://phabricator.wikimedia.org/T174864) [16:45:57] 10Gerrit, 10Patch-For-Review, 10User-Ladsgroup: Make gerrit use the new WMF logo - https://phabricator.wikimedia.org/T174576#3580714 (10demon) >>! In T174576#3580690, @fgiunchedi wrote: > FWIW this is the svg rendering I got on chrome 60 on Debian stretch before the revert during puppet swat today > > {F934... [16:47:05] 10Scap: scap sync failed on i18n - https://phabricator.wikimedia.org/T175041#3580717 (10thcipriani) Hrm, so it's failing at: to run `sudo -u l10nupdate -n -- scap cdb-rebuild --no-progress --staging --verbose` (from here: https://github.com/wikimedia/scap/blob/master/scap/tasks.py#L317-L322) I think this is a... [16:48:25] (03CR) 10Florianschmidtwelzow: Archive WebPlatformSearchAutocomplete (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/375610 (https://phabricator.wikimedia.org/T174869) (owner: 10MarcoAurelio) [16:54:57] 10Release-Engineering-Team (Kanban), 10Release Pipeline (Blubber): Fix or remove Blubber's node_modules optimization - https://phabricator.wikimedia.org/T171632#3580740 (10dduvall) 05Open>03Resolved [16:55:59] matt_flaschen: do you have a link to the l10n change you were trying to sync when you hit T175041 ? Trying to recreate. [16:55:59] T175041: scap sync failed on i18n - https://phabricator.wikimedia.org/T175041 [16:58:44] thcipriani: [16:59:01] 1. https://gerrit.wikimedia.org/r/#/c/376032/ (this is the tricky one, relatively speaking, since it renames message keys, only in i18n and qqq, which means the other languages will fall back to i18n, but this feature is dark-launched and even when it is launched for real, it will first be a beta feature; translatewiki.net will rename them in other languages per https://www.mediawiki.org/wiki/Localisation#Changing_existing_messages) [16:59:07] only in en and qqq [16:59:14] 2. https://gerrit.wikimedia.org/r/#/c/375451/ [16:59:18] 3. https://gerrit.wikimedia.org/r/#/c/374670/ [16:59:20] thcipriani, that's it. [16:59:32] Also, "fall back to English" [17:00:56] thank you, I'll keep digging. We changed a few things in the most recent scap release that went out yesterday, but we didn't touch cdb/l10n updating, so I'm trying to piece together what the problem could be. [17:01:09] thcipriani: computers suck [17:08:19] Reedy: please make stickers or pins with that! [17:08:24] heh [17:09:11] matt_flaschen: The thing about "cannot delete empty directory" is my doing (and a red herring in your stacktrace) [17:09:17] Just finished cleaning up /that/ part at least [17:11:20] no_justification, yeah, I wasn't that worried about that (but is the underlying cause fixed?) but I'm more concerned about "Could not chdir to home directory /var/lib/mwdeploy: No such file or directory" ^ thcipriani [17:11:36] Because if it's starting in the wrong directory, obviously that will cause problems. [17:11:59] Unless it does everything by absolute path. [17:12:01] matt_flaschen: Eh, underlying cause is hard to fix. [17:12:08] * no_justification mutters something about permissions and glares at l10nupdate [17:12:19] OK, would be nice, but I realized that wasn't the main cause. [17:12:33] No biggie just clarifying :) [17:12:46] no_justification, yep, thanks. [17:13:57] 17:12:44 Command failed [cleaning-patches]: rm-fR /srv/patches/1.30.0-wmf.12 [17:14:08] Well, duh. rm-fR is not a valid bash command :\ [17:14:10] Stupid python [17:17:06] AndyRussG: if you updated authenticator settings then does anything need to change? /me isn't sure ... [17:20:26] twentyafterfour: not sure how it works exactly... For GMail I went into the config and there was a barcode that I scanned with my phone to set up the Authenticator on that phone [17:20:37] So in theory I guess I just need to get into my Phab account to do that [17:20:54] oh ok ... [17:20:54] Or maybe get a code I can use to set up the Authenticator app manually [17:21:19] 10Release-Engineering-Team (Kanban), 10Release Pipeline (Blubber), 10Tracking: Complete Blubber's support for multi-stage Dockerfiles - https://phabricator.wikimedia.org/T174620#3580915 (10dduvall) p:05Triage>03Normal [17:23:49] 10Release-Engineering-Team (Kanban), 10Release Pipeline (Blubber): Optimized image builds for node applications - https://phabricator.wikimedia.org/T174626#3580923 (10dduvall) p:05Triage>03Normal [17:25:00] 10Release-Engineering-Team (Kanban), 10Release Pipeline (Blubber): Shorthand for standard multi-stage artifact config - https://phabricator.wikimedia.org/T174622#3580928 (10dduvall) p:05Triage>03Normal [17:28:42] 10Scap: scap sync failed on i18n - https://phabricator.wikimedia.org/T175041#3580615 (10demon) ``` 15:34:43 ['/usr/bin/scap', 'pull-master', 'tin.eqiad.wmnet'] on naos.codfw.wmnet returned [70]: Could not chdir to home directory /var/lib/mwdeploy: No such file or directory ``` We really should get around to fix... [17:33:47] matt_flaschen: no_justification I'm still not sure what the underlying cause of the problem is. There are a few red herrings in the error output (could not chdir/cannot delete empty directory) but I think the root cause is that it couldn't create the tmp cdb file, but I don't know *why* yet :( [17:40:46] thcipriani, did you check for disk space? It's not the first alphabetical, but maybe it's not in alphabetical order. [17:42:15] yeah, all seems fine there (we're using like 5%), but that was a good thought :) [17:43:48] Yeah [17:44:04] I'm trying to whack-a-mole the red herrings too [17:47:21] 10Release-Engineering-Team (Kanban), 10Release Pipeline (Blubber): Optimized image builds for node applications - https://phabricator.wikimedia.org/T174626#3581059 (10dduvall) [17:48:01] (03PS2) 10Jdlrobson: Add a Jenkins job for Popups browser tests [integration/config] - 10https://gerrit.wikimedia.org/r/375377 (https://phabricator.wikimedia.org/T174786) [17:48:13] 10Release-Engineering-Team, 10Page-Previews, 10Performance-Team, 10Readers-Web-Backlog, 10User-zeljkofilipin: Provide a reliable test environment that mimics production for running integration tests - https://phabricator.wikimedia.org/T174786#3581062 (10Jdlrobson) [17:55:43] (03CR) 10Legoktm: Add sniff to find unused "use" statements (031 comment) [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/375547 (https://phabricator.wikimedia.org/T167694) (owner: 10Legoktm) [17:57:32] (03PS4) 10Legoktm: Add sniff to find unused "use" statements [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/375547 (https://phabricator.wikimedia.org/T167694) [18:00:10] 10Scap: scap sync failed on i18n - https://phabricator.wikimedia.org/T175041#3581111 (10thcipriani) So there a few items in the output that are red herrings: ``` ​cannot delete non-empty directory: php-1.30.0-wmf.14/cache/l10n ``` This happens because there are `.cdb` files under `php-1.30.0-wmf.14/cache/l10n`... [18:06:48] (03CR) 10jerkins-bot: [V: 04-1] Add sniff to find unused "use" statements [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/375547 (https://phabricator.wikimedia.org/T167694) (owner: 10Legoktm) [18:07:44] (03PS5) 10Legoktm: Add sniff to find unused "use" statements [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/375547 (https://phabricator.wikimedia.org/T167694) [18:16:53] (03PS6) 10Legoktm: Add sniff to find unused "use" statements [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/375547 (https://phabricator.wikimedia.org/T167694) [18:30:16] Hi all... Mmm got another releng-y question here... Is browserstack.com the place I could go to debug stuff on an iPad emulator? Pretty sure that's what I've used before for similar stuff. They have the wp logo on their homepage, but when I log in, it only gives me the "free trial" options, in which the iPad emulators don't give access to Safari dev tools.... [18:33:17] AndyRussG: you need to talk with adam baso about that (him and sarah r manage that service subscription) [18:33:39] we have a limited number of slots/accounts (10 I think) [18:39:20] AndyRussG, there is also saucelabs.com , which we have official access to as well (and supports various iPads). [18:39:26] greg-g: K thanks... Mmm actually just was able to reproduce the bug on the iPhone simulator, which does provide dev tools for free. So I'll try to continue using that, and will reach out if it doesn't work or if I run out of free minutes... Thx again! [18:39:39] matt_flaschen: ahhh right yeah I was thinking there was another one [18:39:54] matt_flaschen: thx :) [18:40:01] AndyRussG, saucelabs also lets you test localhost. (The other may as well, don't know.) [18:44:22] matt_flaschen: cool! yeah BrowserStack offered to, also, via an extension... didn't try it tho [18:44:46] 10Scap: scap sync failed on i18n - https://phabricator.wikimedia.org/T175041#3581251 (10thcipriani) I think the problem was caused by (from SAL): ``` 15:36 mattflaschen@tin: scap aborted: Prepare to enable RCFilters (WLFilters) on Watchlist (duration: 08m 42s) 15:35 marostegui@tin: Synchronized wmf-config/db-eq... [18:47:14] thcipriani, sync-file touches i18n? My understanding was that sync-file and sync-dir don't touch i18n at all. [18:49:45] matt_flaschen: every sync-* operation triggers a sync to the co-master, so that way if we lose tin or naos unexpectedly /srv/mediawiki-staging will be the same on both, so every sync operation calls: https://github.com/wikimedia/scap/blob/master/scap/tasks.py#L290-L322 [18:50:03] so this is for masters in /srv/mediawiki-staging only [18:50:41] tl;dr: that function just runs rsync from active-master -> passive master and then rebuilds cdbs from json on the masters [18:50:44] 10Continuous-Integration-Config, 10Release-Engineering-Team, 10Operations: operations-puppet-tests-docker console output lacks color - https://phabricator.wikimedia.org/T175057#3581282 (10hashar) [18:50:59] *rebuilds cdbs from json on the passive master [18:52:48] thcipriani, thanks, I didn't know that. [18:52:59] thcipriani, also, my bad for going over my window (I did note in #wikimedia-operations that I was going to go over (there was no window in the next hour), and I even saw that sync-file, but I didn't think that was the issue). [18:53:01] 10Continuous-Integration-Config, 10Release-Engineering-Team, 10Operations: operations-puppet-tests-docker console output lacks color - https://phabricator.wikimedia.org/T175057#3581296 (10hashar) From https://groups.google.com/forum/#!topic/docker-user/Bp4BaWRw6k4 > No colours: > `docker run -v ~/myproject/... [18:54:12] thcipriani, that seems like a performance penalty for sync-file and sync-dir (which are used by definition when there is no i18n change). Would it be enough to sync the CDBs to the co-master as the last step of scap sync? These are derived files anyway. [18:56:45] it is a little bit of a performance penalty, but there is logic that prevents them from being rebuilt unless mtimes have changed. Basically, the penalty is a stat call for sync-file and sync-dir rather than a rebuild; however, I think in this instance the full scap invalidated mtimes for that file during the other sync. [18:57:13] and so sync-file went ahead and tried to rebuild which messed up the full scap [18:57:42] we have lockfile logic that should prevent two sync-* operations from runnning in parallel, but that failed here :( [19:00:52] matt_flaschen: Nitpick: sync-dir and sync-file are synonymous :) [19:04:31] 10Gerrit, 10Patch-For-Review, 10User-Ladsgroup: Make gerrit use the new WMF logo - https://phabricator.wikimedia.org/T174576#3581402 (10Ladsgroup) As I said in the gerrit patch, there is no designer, resource or plans defined for designing a brand-new logo, thus it's not an alternative to my patch. The new l... [19:10:55] 10Gerrit, 10Patch-For-Review, 10User-Ladsgroup: Make gerrit use the new WMF logo - https://phabricator.wikimedia.org/T174576#3581456 (10Legoktm) I asked @isarra on IRC and she said to remind her next week about designing a new log. [19:23:13] (03CR) 10Legoktm: [C: 032] "No diff in job :)" [integration/config] - 10https://gerrit.wikimedia.org/r/376011 (owner: 10Hashar) [19:24:23] (03Merged) 10jenkins-bot: remexhtml publish: use YAML aliasing (DRY) [integration/config] - 10https://gerrit.wikimedia.org/r/376011 (owner: 10Hashar) [19:31:54] is this a known bug? "Catchable fatal error: Argument 4 passed to MediaWiki\Linker\LinkRenderer::makeKnownLink() must be an instance of array, string given in /srv/mediawiki/php-1.30.0-wmf.16/extensions/FlaggedRevs/frontend/specialpages/reports/ReviewedPages_body.php on line 1" [19:32:05] saw in fatal monitor [19:34:42] Amir1: 3269e8db73dcb2d5f60060d7eacad46cdb94f291 I think [19:34:42] 10Gerrit, 10Patch-For-Review, 10User-Ladsgroup: Make gerrit use the new WMF logo - https://phabricator.wikimedia.org/T174576#3581625 (10demon) >>! In T174576#3581402, @Ladsgroup wrote: > As I said in the gerrit patch, there is no designer, resource or plans defined for designing a brand-new logo, I think yo... [19:36:01] hm, we should have backported that [19:37:07] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Watching / External), 10Cloud-VPS, 10Nodepool, 10cloud-services-team (Kanban): rabbitmq: Consume and log messages sent to notifications.error - https://phabricator.wikimedia.org/T175029#3581634 (10hashar) [19:49:19] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Watching / External), 10Cloud-VPS, 10Nodepool, 10cloud-services-team (Kanban): rabbitmq: Consume and log messages sent to notifications.error - https://phabricator.wikimedia.org/T175029#3581733 (10chasemp) should we try to get this into... [19:59:54] 10Scap: scap sync failed on i18n - https://phabricator.wikimedia.org/T175041#3581778 (10thcipriani) [20:05:22] 10Scap: scap sync failed on i18n - https://phabricator.wikimedia.org/T175041#3581800 (10thcipriani) The core issue here is that scap locking failed and allowed two `sync*` processes to happen at the same time. This caused some weird behavior as there are some race conditions within the deployment since it is not... [20:08:02] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Watching / External), 10Cloud-VPS, 10Nodepool, and 2 others: figure out if nodepool is overwhelming rabbitmq and/or nova - https://phabricator.wikimedia.org/T170492#3581822 (10hashar) Also from the logs there are `ValueError: Circular ref... [20:09:34] 10Continuous-Integration-Config, 10MediaWiki-extensions-Scribunto: Run Scribunto test suite on luasandbox patches - https://phabricator.wikimedia.org/T175068#3581826 (10Legoktm) [20:26:14] 10Release-Engineering-Team (Watching / External), 10Operations, 10ops-eqiad: tin has a failing hdd - https://phabricator.wikimedia.org/T174449#3581919 (10Peachey88) [20:46:00] (03CR) 10MarcoAurelio: Archive WebPlatformSearchAutocomplete (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/375610 (https://phabricator.wikimedia.org/T174869) (owner: 10MarcoAurelio) [20:56:54] 10Release-Engineering-Team (Watching / External), 10Wikidata: Decide what to do with Wikibase JS-only libraries regarding the build/deployment of Wikidata code - https://phabricator.wikimedia.org/T174922#3582105 (10greg) [21:48:25] 10Release-Engineering-Team, 10Page-Previews, 10Performance-Team, 10Readers-Web-Backlog, 10User-zeljkofilipin: Provide a reliable test environment that mimics production for running integration tests - https://phabricator.wikimedia.org/T174786#3573043 (10greg) Thanks @krinkle for laying out all that backg... [22:01:26] 10Release-Engineering-Team (Kanban), 10Phabricator, 10Scoring-platform-team: Migrate #Scoring-platform-team to be a phab milestone - https://phabricator.wikimedia.org/T171513#3582352 (10mmodell) a:03mmodell @ksmith: Indeed it is. [22:05:00] Project beta-update-databases-eqiad build #19718: 15ABORTED in 45 min: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/19718/ [22:08:05] 10Release-Engineering-Team (Kanban), 10Phabricator, 10Scoring-platform-team: Migrate #Scoring-platform-team to be a phab milestone - https://phabricator.wikimedia.org/T171513#3467572 (10awight) I like it--the never-ending milestone ;-) https://i.pinimg.com/736x/0a/19/a9/0a19a950cc33a64c5b4d3d2d006e5193--char... [22:08:21] 10Release-Engineering-Team (Kanban), 10Release, 10Train Deployments: 1.30.0-wmf.17 deployment blockers - https://phabricator.wikimedia.org/T170635#3582381 (10demon) [22:08:23] 10Release-Engineering-Team (Kanban), 10Release, 10Train Deployments: 1.30.0-wmf.17 deployment blockers - https://phabricator.wikimedia.org/T170635#3437656 (10demon) [22:17:20] 10Gerrit, 10Release-Engineering-Team (Kanban), 10Cleanup, 10utfnormal: Archive php-normal which is unused - https://phabricator.wikimedia.org/T174583#3582404 (10demon) 05Open>03Resolved Done [22:26:45] (03CR) 10Thcipriani: [C: 032] Phase out legacy operations/puppet jobs [integration/config] - 10https://gerrit.wikimedia.org/r/376034 (owner: 10Hashar) [22:29:14] (03Abandoned) 10Awight: Run DonationInterface LintYaml job via composer-test [integration/config] - 10https://gerrit.wikimedia.org/r/299185 (owner: 10Awight) [22:29:17] (03Merged) 10jenkins-bot: Phase out legacy operations/puppet jobs [integration/config] - 10https://gerrit.wikimedia.org/r/376034 (owner: 10Hashar) [22:30:52] 10Gerrit: Blue button to do Verified+2 - https://phabricator.wikimedia.org/T175090#3582475 (10Mattflaschen-WMF) [22:49:41] 10Gerrit: Blue button to do Verified+2 - https://phabricator.wikimedia.org/T175090#3582475 (10demon) Then revoke permissions from people being able to V+2 on said repo (many already do) Changing this behavior would be...nontrivial.... [23:03:35] !log reloading zuul to deploy https://gerrit.wikimedia.org/r/#/c/376034 [23:03:39] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [23:05:00] Project beta-update-databases-eqiad build #19719: 15ABORTED in 45 min: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/19719/ [23:12:03] 10Continuous-Integration-Config, 10Release-Engineering-Team (Backlog), 10Operations: operations-puppet-tests-docker console output lacks color - https://phabricator.wikimedia.org/T175057#3582656 (10greg) [23:29:33] 10Release-Engineering-Team (Kanban), 10Release Pipeline (Blubber): Shorthand for standard multi-stage artifact config - https://phabricator.wikimedia.org/T174622#3582704 (10dduvall) [23:33:20] 10Release-Engineering-Team (Kanban), 10Release Pipeline (Blubber), 10Tracking: Ensure application files are not copied for final multi-stage images - https://phabricator.wikimedia.org/T174623#3582714 (10dduvall) p:05Triage>03Normal