[00:03:35] PROBLEM - Puppet errors on integration-r-lang-01 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [00:43:35] RECOVERY - Puppet errors on integration-r-lang-01 is OK: OK: Less than 1.00% above the threshold [0.0] [01:04:34] PROBLEM - Puppet errors on integration-r-lang-01 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [01:39:34] RECOVERY - Puppet errors on integration-r-lang-01 is OK: OK: Less than 1.00% above the threshold [0.0] [02:13:41] Project beta-scap-eqiad build #168788: 04FAILURE in 0.47 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/168788/ [02:26:05] Yippee, build fixed! [02:26:05] Project beta-scap-eqiad build #168789: 09FIXED in 2 min 27 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/168789/ [03:57:32] 10Release-Engineering-Team (Kanban), 10Release, 10Train Deployments: 1.30.0-wmf.14 deployment blockers - https://phabricator.wikimedia.org/T170632#3437602 (10Legoktm) https://gerrit.wikimedia.org/r/#/c/372097/ is a cherry-pick for {T173394}. [04:42:25] 10Release-Engineering-Team, 10Wikimedia-Site-requests: Remove zerowiki from group0 wikis - https://phabricator.wikimedia.org/T173397#3526961 (10Legoktm) [04:56:47] PROBLEM - Mediawiki Error Rate on graphite-labs is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [10.0] [05:03:53] (03CR) 10Bekicot: Fix Rubocop Offenses (033 comments) [ruby/api] - 10https://gerrit.wikimedia.org/r/368609 (owner: 10Bekicot) [05:07:04] (03CR) 10Bekicot: "> Patch set #4 fixes whitespace in client.rb, but not in" [ruby/api] - 10https://gerrit.wikimedia.org/r/368609 (owner: 10Bekicot) [05:46:51] PROBLEM - Mediawiki Error Rate on graphite-labs is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [10.0] [06:21:49] PROBLEM - Mediawiki Error Rate on graphite-labs is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [10.0] [06:35:35] PROBLEM - Puppet errors on integration-r-lang-01 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [06:42:50] 10MediaWiki-Releasing, 10Release-Engineering-Team (Kanban), 10MW-1.29-release-notes, 10Patch-For-Review: Include release extensions/skins/vendor as submodules of core - https://phabricator.wikimedia.org/T137564#3527018 (10Legoktm) [06:42:52] 10Release-Engineering-Team, 10MW-1.29-release, 10MW-1.29-release-notes, 10Patch-For-Review, 10Release: MediaWiki 1.29 tarball comes with the wrong extensions - and misses some - https://phabricator.wikimedia.org/T171197#3527017 (10Legoktm) 05Resolved>03Open [06:54:26] PROBLEM - Puppet errors on integration-slave-jessie-1001 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [06:58:29] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban), 10Performance-Team, 10WebPageTest: Where to trigger WebPageTest jobs? - https://phabricator.wikimedia.org/T166756#3527020 (10Peter) I've increased to two build executors through the Jenkins GUI, could see that the two jobs collid... [07:10:36] RECOVERY - Puppet errors on integration-r-lang-01 is OK: OK: Less than 1.00% above the threshold [0.0] [07:13:39] PROBLEM - Puppet errors on deployment-sentry01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [07:34:26] RECOVERY - Puppet errors on integration-slave-jessie-1001 is OK: OK: Less than 1.00% above the threshold [0.0] [08:56:49] RECOVERY - Mediawiki Error Rate on graphite-labs is OK: OK: Less than 1.00% above the threshold [1.0] [09:06:34] PROBLEM - Puppet errors on integration-r-lang-01 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [09:41:38] RECOVERY - Puppet errors on integration-r-lang-01 is OK: OK: Less than 1.00% above the threshold [0.0] [10:57:46] PROBLEM - Free space - all mounts on deployment-jobrunner02 is CRITICAL: CRITICAL: deployment-prep.deployment-jobrunner02.diskspace.root.byte_percentfree (<100.00%) [12:02:34] PROBLEM - Puppet errors on integration-r-lang-01 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [12:42:36] RECOVERY - Puppet errors on integration-r-lang-01 is OK: OK: Less than 1.00% above the threshold [0.0] [13:03:35] PROBLEM - Puppet errors on integration-r-lang-01 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [13:45:40] Yippee, build fixed! [13:45:41] Project selenium-MinervaNeue » chrome,beta,Linux,BrowserTests build #78: 09FIXED in 21 min: https://integration.wikimedia.org/ci/job/selenium-MinervaNeue/BROWSER=chrome,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=BrowserTests/78/ [14:13:37] RECOVERY - Puppet errors on integration-r-lang-01 is OK: OK: Less than 1.00% above the threshold [0.0] [14:55:50] (03CR) 10Zfilipin: Fix Rubocop Offenses (031 comment) [ruby/api] - 10https://gerrit.wikimedia.org/r/368609 (owner: 10Bekicot) [14:57:32] (03CR) 10Zfilipin: [C: 032] Fix Rubocop Offenses [ruby/api] - 10https://gerrit.wikimedia.org/r/368609 (owner: 10Bekicot) [14:58:08] (03Merged) 10jenkins-bot: Fix Rubocop Offenses [ruby/api] - 10https://gerrit.wikimedia.org/r/368609 (owner: 10Bekicot) [14:58:40] (03CR) 10jenkins-bot: Fix Rubocop Offenses [ruby/api] - 10https://gerrit.wikimedia.org/r/368609 (owner: 10Bekicot) [15:36:08] (03CR) 10Thcipriani: [C: 031] "One inline nit, but overall looks good." (032 comments) [integration/config] - 10https://gerrit.wikimedia.org/r/371708 (owner: 10Addshore) [16:36:28] 10Release-Engineering-Team (Kanban), 10Phabricator: Phabricator data dump hasn't run automatically since the Feb 17 2016 upgrade. - https://phabricator.wikimedia.org/T127524#3528089 (10Dzahn) >>! In T127524#2087788, @ArielGlenn wrote: > I see cron output today: > NOTICE: rtppl not found! > > Is this an issu... [16:58:05] 10Release-Engineering-Team (Kanban), 10Phabricator: Phabricator data dump hasn't run automatically since the Feb 17 2016 upgrade. - https://phabricator.wikimedia.org/T127524#3528169 (10Aklapper) >>! In T127524#3528087, @Dzahn wrote: > We are seeing these again. cron mail with the "NOTICE: rtppl not found!" me... [16:58:26] 10Continuous-Integration-Config, 10Scoring-platform-team-Backlog: Have CI merge research/ores/wheels changes - https://phabricator.wikimedia.org/T173251#3522157 (10Halfak) p:05Triage>03Normal [16:59:14] 10Continuous-Integration-Config, 10Scoring-platform-team-Backlog, 10Easy, 10User-Ladsgroup: Have CI merge research/ores/wheels changes - https://phabricator.wikimedia.org/T173251#3528178 (10Halfak) a:03Ladsgroup [17:00:29] 10Scap (Scap3-Adoption-Phase1), 10Analytics, 10Analytics-EventLogging, 10Performance-Team: Use scap3 to deploy eventlogging/eventlogging - https://phabricator.wikimedia.org/T118772#3528180 (10greg) [17:00:31] 10Scap (Scap3-Adoption-Phase1), 10releng-201516-q4, 10releng-201718-q1, 10Trebuchet: [keyresult] Migrate remaining trebuchet deployed services - https://phabricator.wikimedia.org/T129290#3528179 (10greg) [17:01:41] (03CR) 10Umherirrender: [C: 032] Add sniff to ensure floats have a leading `0` if necessary [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/371986 (https://phabricator.wikimedia.org/T170442) (owner: 10Legoktm) [17:02:20] 10MediaWiki-Codesniffer, 10Patch-For-Review: Always use leading zeros in php files for floats - https://phabricator.wikimedia.org/T170442#3528197 (10Umherirrender) 05Open>03Resolved p:05Triage>03Normal [17:02:31] (03Merged) 10jenkins-bot: Add sniff to ensure floats have a leading `0` if necessary [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/371986 (https://phabricator.wikimedia.org/T170442) (owner: 10Legoktm) [17:14:50] 10Beta-Cluster-Infrastructure, 10ORES, 10Scoring-platform-team-Backlog: On beta cluster, ORESFetchScoreJob got a HTTP 400 bad request from ores-beta - https://phabricator.wikimedia.org/T157790#3528247 (10Halfak) [17:19:29] 10Release-Engineering-Team (Kanban), 10Release, 10Train Deployments: 1.30.0-wmf.14 deployment blockers - https://phabricator.wikimedia.org/T170632#3528265 (10thcipriani) [17:20:26] 10Release-Engineering-Team (Kanban), 10Release, 10Train Deployments: 1.30.0-wmf.14 deployment blockers - https://phabricator.wikimedia.org/T170632#3437602 (10demon) >>! In T170632#3526952, @Legoktm wrote: > https://gerrit.wikimedia.org/r/#/c/372097/ is a cherry-pick for {T173394}. Merged and deployed. Thanks! [17:28:47] 10Release-Engineering-Team (Kanban): 20170721-Train-Wikidata - Post mortem - https://phabricator.wikimedia.org/T173435#3528319 (10Jrbranaa) [17:29:45] 10Release-Engineering-Team (Kanban): 20170721-Train-Wikidata - Post mortem - https://phabricator.wikimedia.org/T173435#3528334 (10greg) [17:34:37] PROBLEM - Puppet errors on integration-r-lang-01 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [17:44:30] 10Release-Engineering-Team (Kanban), 10Release, 10Train Deployments: MW 1.30.0-wmf.10 deployment blockers - https://phabricator.wikimedia.org/T168050#3528391 (10greg) [17:44:43] 10Deployment-Systems, 10Scap (Scap3-Adoption-Phase1), 10scap2, 10OCG-PDFRenderer: Deploy ocg with scap3 - https://phabricator.wikimedia.org/T129142#2096453 (10Pchelolo) The plan is to sunset OCG, so it's doesn't make sense to put effort in migrating it T150871 [17:48:56] 10Deployment-Systems, 10Scap (Scap3-Adoption-Phase1), 10scap2, 10OCG-PDFRenderer: Deploy ocg with scap3 - https://phabricator.wikimedia.org/T129142#3528429 (10greg) >>! In T129142#3528392, @Pchelolo wrote: > The plan is to sunset OCG, so it's doesn't make sense to put effort in migrating it T150871 How so... [17:51:29] 10Deployment-Systems, 10Scap (Scap3-Adoption-Phase1), 10scap2, 10OCG-PDFRenderer, 10Services (watching): Deploy ocg with scap3 - https://phabricator.wikimedia.org/T129142#3528433 (10Pchelolo) That idea was around for quite a while, but since there's no clear owner of OCG it was just dragging. I think @GW... [17:58:54] 10Deployment-Systems, 10Scap (Scap3-Adoption-Phase1), 10scap2, 10OCG-PDFRenderer, 10Services (watching): Deploy ocg with scap3 - https://phabricator.wikimedia.org/T129142#3528444 (10GWicke) See {T150871}. Original timeline / agreement with Reading was to sunset OCG this month. This has not happened yet,... [17:59:06] 10Deployment-Systems, 10Scap (Scap3-Adoption-Phase1), 10scap2, 10OCG-PDFRenderer, 10Services (watching): Deploy ocg with scap3 - https://phabricator.wikimedia.org/T129142#3528445 (10greg) There is a working group (that I am on, that we just made on Monday, of tech-mgt people) to work through what it mean... [17:59:57] 10Deployment-Systems, 10Scap (Scap3-Adoption-Phase1), 10scap2, 10OCG-PDFRenderer, 10Services (watching): Deploy ocg with scap3 - https://phabricator.wikimedia.org/T129142#3528446 (10greg) Ah, useful, thanks @gwicke ! I'll follow that task. [18:00:56] 10Deployment-Systems, 10Scap (Scap3-Adoption-Phase1), 10scap2, 10OCG-PDFRenderer, 10Services (watching): Deploy ocg with scap3 - https://phabricator.wikimedia.org/T129142#3528447 (10mobrovac) >>! In T129142#3528429, @greg wrote: > How soon, by end of quarter? That's the deadline for trebuchet removals.... [18:03:48] 10Deployment-Systems, 10Scap (Scap3-Adoption-Phase1), 10scap2, 10OCG-PDFRenderer, 10Services (watching): Deploy ocg with scap3 - https://phabricator.wikimedia.org/T129142#2096453 (10greg) Thanks @mobrovac, I poked a little over on the sunsetting epic gwicke mentioned. [18:09:35] RECOVERY - Puppet errors on integration-r-lang-01 is OK: OK: Less than 1.00% above the threshold [0.0] [18:41:09] hi. after my access was revoked i am back in wikitech and phab and gmail, just not gerrit [18:41:34] i have a new password on wikitech and 2fa and it all works, but Gerrit doesnt like it [18:42:44] i have already waited over night for sync etc [19:03:58] 10Release-Engineering-Team (Watching / External), 10JobRunner-Service, 10Operations, 10Performance-Team, and 2 others: Collect error logs from jobchron/jobrunner services in Logstash - https://phabricator.wikimedia.org/T172479#3528746 (10Gilles) p:05Triage>03Low [19:04:53] 10Release-Engineering-Team (Watching / External), 10JobRunner-Service, 10Operations, 10Performance-Team, and 2 others: Collect error logs from jobchron/jobrunner services in Logstash - https://phabricator.wikimedia.org/T172479#3528748 (10Krinkle) >>! In T172479#3502395, @greg wrote: > Adding ~~`#mediawiki-... [19:04:53] it looks my account is just set to inactive. bawolff is looking to reactivate it [19:09:56] Project selenium-MinervaNeue » chrome,beta,Linux,BrowserTests build #79: 04FAILURE in 20 min: https://integration.wikimedia.org/ci/job/selenium-MinervaNeue/BROWSER=chrome,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=BrowserTests/79/ [19:14:13] 10Deployment-Systems, 10MediaWiki-extensions-CentralAuth, 10Performance-Team: $wgLocalVirtualHosts should include login.wikimedia.org, wikidata.org and others? - https://phabricator.wikimedia.org/T172357#3528789 (10Gilles) p:05Triage>03Low a:03Krinkle [19:20:57] (03CR) 10Thcipriani: [C: 032] Update the php symlink after group1 promote [tools/release] - 10https://gerrit.wikimedia.org/r/364833 (owner: 10Thcipriani) [19:21:35] (03Merged) 10jenkins-bot: Update the php symlink after group1 promote [tools/release] - 10https://gerrit.wikimedia.org/r/364833 (owner: 10Thcipriani) [19:21:37] 10Deployment-Systems, 10MediaWiki-extensions-CentralAuth, 10Performance-Team: $wgLocalVirtualHosts should include login.wikimedia.org, wikidata.org and others? - https://phabricator.wikimedia.org/T172357#3528799 (10aaron) Another option is to have the MediaWiki.php code use InterWikiLookup::getAllPrefixes()... [19:21:55] ok, back on gerrit. resolved [19:23:13] (03PS1) 10Umherirrender: Remove unmatched @codingStandardsIgnoreEnd [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/372176 [19:28:34] 10Release-Engineering-Team (Kanban), 10Phabricator: Phabricator data dump hasn't run automatically since the Feb 17 2016 upgrade. - https://phabricator.wikimedia.org/T127524#3528824 (10Dzahn) >>! In T127524#3528169, @Aklapper wrote: > (Also see {T172558} for recent issues with the dump file.) Yea, but there i... [19:36:10] (03CR) 10Krinkle: [C: 032] Remove unmatched @codingStandardsIgnoreEnd [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/372176 (owner: 10Umherirrender) [19:36:56] (03Merged) 10jenkins-bot: Remove unmatched @codingStandardsIgnoreEnd [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/372176 (owner: 10Umherirrender) [19:41:24] 10Release-Engineering-Team (Kanban), 10Phabricator: Phabricator data dump hasn't run automatically since the Feb 17 2016 upgrade. - https://phabricator.wikimedia.org/T127524#3528857 (10Dzahn) this also appears here: https://www.mediawiki.org/wiki/Phabricator/Trello_to_Phabricator/Research_import#Debug_output... [19:42:05] PROBLEM - Puppet errors on deployment-mathoid is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [19:49:15] (03PS1) 10Umherirrender: Whitelist second email of Kghbln [integration/config] - 10https://gerrit.wikimedia.org/r/372181 [19:50:51] 10Release-Engineering-Team (Backlog), 10Phabricator, 10Performance-Team (Radar): Custom domain/URL for phame performance blog - https://phabricator.wikimedia.org/T166374#3528925 (10Krinkle) 05Open>03Resolved a:03Krinkle Short url seems fine for now.... [19:52:29] (03PS2) 10Umherirrender: Whitelist second email of Kghbln [integration/config] - 10https://gerrit.wikimedia.org/r/372181 [19:52:56] (03PS3) 10Umherirrender: Whitelist second email of Kghbln [integration/config] - 10https://gerrit.wikimedia.org/r/372181 [19:52:59] 10Release-Engineering-Team (Kanban), 10Phabricator: Phabricator data dump hasn't run automatically since the Feb 17 2016 upgrade. - https://phabricator.wikimedia.org/T127524#3528942 (10Dzahn) @chasemp ^ In https://github.com/wikimedia/phabricator-tools/blob/master/wmfphablib/rtlib.py can't find `rtppl `in "`... [19:53:34] Hm.. I just dragged a task on the boardboard of performance-team from a sub-project column "Radar" to regular column "Blocked" - https://phabricator.wikimedia.org/T161192 [19:53:38] And now it's in both? [20:00:18] mutante, hey [20:00:40] mutante, did you try SSH to gerrit? [20:00:52] as in, ssh -p 29418 dzahn@gerrit.wikimedia.org [20:01:07] Krenair: i did not, but Bawolff fixed my web login [20:01:17] my user was set to inactive [20:01:20] ah [20:01:26] yes that might happen when your access gets revoked [20:01:51] yea, as it should be. i just had to request getting it back [20:02:03] and proof i have wikitech access / GPG signed [20:02:14] and now i have it back. thanks for checking [20:02:21] yeah, sounds like someone missed it when putting you back [20:02:45] i think it was simpler than that.. nothing triggered "put me back" :) [20:03:06] i just got new hardware 24h ago [20:08:33] 10Release-Engineering-Team (Kanban), 10Phabricator: Phabricator data dump hasn't run automatically since the Feb 17 2016 upgrade. - https://phabricator.wikimedia.org/T127524#3528995 (10chasemp) >>! In T127524#3528942, @Dzahn wrote: > @chasemp ^ In https://github.com/wikimedia/phabricator-tools/blob/master/wmfp... [20:22:07] RECOVERY - Puppet errors on deployment-mathoid is OK: OK: Less than 1.00% above the threshold [0.0] [20:22:59] Yippee, build fixed! [20:22:59] Project selenium-MinervaNeue » chrome,beta,Linux,BrowserTests build #80: 09FIXED in 21 min: https://integration.wikimedia.org/ci/job/selenium-MinervaNeue/BROWSER=chrome,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=BrowserTests/80/ [20:46:38] 10Browser-Tests-Infrastructure, 10MediaWiki-extensions-CentralAuth, 10VisualEditor, 10Browser-Tests, 10Readers-Web-Backlog (Tracking): Edits sometimes fail for new users erroneously reporting them as anon - https://phabricator.wikimedia.org/T172835#3529110 (10Jdlrobson) [21:15:56] RECOVERY - Puppet errors on deployment-etcd-01 is OK: OK: Less than 1.00% above the threshold [0.0] [21:16:49] 10Release-Engineering-Team (Kanban), 10Release, 10Train Deployments: 1.30.0-wmf.14 deployment blockers - https://phabricator.wikimedia.org/T170632#3529198 (10thcipriani) [21:18:51] so I just got access to deployment-prep again [21:19:10] thought I'd check in on how puppet is doing, some instances have problems [21:19:17] as you saw above, -etcd-01 is fixed [21:19:23] -sca02 login is broken [21:21:33] fixed -sca02 login by restarting nslcd service [21:22:54] (-sca02 puppet is also fixed now, error was due to missing 'prometheus' user which was broken by the same problem with nslcd) [21:30:04] -eventlogging04 puppet fixed by creating /var/lib/scap dir [21:34:07] PROBLEM - Puppet errors on deployment-kafka01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [21:34:29] RECOVERY - Puppet errors on deployment-sca02 is OK: OK: Less than 1.00% above the threshold [0.0] [21:36:48] PROBLEM - Puppet errors on deployment-pdf01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [21:38:17] bd808: matt_flaschen: I'm getting really strange errors on Vagrant (in trusty-compat): APT says the git package depends on libpcre2-8-0 which doesn't exist (any more?) [21:38:22] libpcre3-dev is installed though [21:38:32] don't think I can fix -pdf01 without either puppet hieradata patches or admin actions [21:39:22] This breaks provisioning so now I can't add new extensions [21:39:24] -jobrunner02's disk is full [21:39:29] I guess I could destroy and rebuild with jessie? [21:39:30] RECOVERY - Puppet errors on deployment-eventlogging04 is OK: OK: Less than 1.00% above the threshold [0.0] [21:46:31] 10Continuous-Integration-Config, 10MinervaNeue, 10Readers-Web-Backlog, 10Readers-Web-Kanban-Board: MinervaNeue mwext-doxygen-publish fails - https://phabricator.wikimedia.org/T173255#3529290 (10pmiazga) [21:46:58] -rw-r----- 1 root wikidev 11G Aug 16 21:46 /var/log/mediawiki/jobrunner.log [21:47:01] that's not a small log [21:48:10] addgoat, ^ [21:49:22] 10Continuous-Integration-Config, 10MinervaNeue, 10Readers-Web-Backlog, 10Readers-Web-Kanban-Board: MinervaNeue mwext-doxygen-publish fails - https://phabricator.wikimedia.org/T173255#3522244 (10pmiazga) Bringing it into the current sprint as development work is done. I reviewed the change and it looked goo... [21:50:33] Krenair: that's sounds big, why is it big? Is it my fault? I fly home tonight [21:50:54] no idea yet [21:50:59] I just looked at the last entry in there [21:51:01] and found this: [21:51:08] Aug 14 18:09:39 deployment-jobrunner02 jobrunner[26763]: curl -XPOST -s -a 'http://127.0.0.1:9005/rpc/RunJobs.php?wiki=dewikivoyage&type=ChangeNotification&maxtime=30&maxmem=300M' STDOUT: [21:51:48] which of course is followed by an error from multiversion about how there is no such wiki in beta [21:51:52] but it's not getting spammed, so [21:59:34] (03PS3) 10MarcoAurelio: Archive Extension:AdvancedMeta [integration/config] - 10https://gerrit.wikimedia.org/r/371640 (https://phabricator.wikimedia.org/T173184) [21:59:43] (03PS3) 10MarcoAurelio: Archive Extension:ImageTagging [integration/config] - 10https://gerrit.wikimedia.org/r/371653 (https://phabricator.wikimedia.org/T167897) [22:00:14] who can +2 these ^^ [22:33:35] 10Scap (Scap3-Adoption-Phase1), 10Analytics, 10Analytics-EventLogging, 10Performance-Team: Use scap3 to deploy eventlogging/eventlogging - https://phabricator.wikimedia.org/T118772#3529414 (10Krinkle) Discussed this today. Given we're only using the eventlogging library as simple abstraction around python-... [22:43:57] 10Beta-Cluster-Infrastructure: Non-existent wiki urls on beta cluster gives Unsafe/Insecure connection message - https://phabricator.wikimedia.org/T173469#3529447 (10Legoktm) [22:53:52] PROBLEM - Puppet errors on deployment-mira is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [23:08:17] 10Release-Engineering-Team (Kanban), 10Phabricator: Phabricator data dump hasn't run automatically since the Feb 17 2016 upgrade. - https://phabricator.wikimedia.org/T127524#3529478 (10Dzahn) Thank you very much chasemp. The goal is just to silence the cron job one way or another then. [23:16:31] 10Beta-Cluster-Infrastructure, 10Upstream: Non-existent wiki urls on beta cluster gives Unsafe/Insecure connection message - https://phabricator.wikimedia.org/T173469#3529501 (10Krenair) We only got certs to cover the existing ones, but: https://letsencrypt.org/2017/07/06/wildcard-certificates-coming-jan-2018.... [23:31:07] 10Beta-Cluster-Infrastructure, 10Upstream: Non-existent wiki urls on beta cluster gives Unsafe/Insecure connection message - https://phabricator.wikimedia.org/T173469#3529562 (10Krenair) Note that I think the above would rely on DNS-01 authentication to run automatically, which would likely involve writing som... [23:33:52] RECOVERY - Puppet errors on deployment-mira is OK: OK: Less than 1.00% above the threshold [0.0] [23:36:05] (03PS1) 10Legoktm: Add config for labs/libraryupgrader [integration/config] - 10https://gerrit.wikimedia.org/r/372208 [23:37:32] (03PS4) 10Legoktm: Archive Extension:ImageTagging [integration/config] - 10https://gerrit.wikimedia.org/r/371653 (https://phabricator.wikimedia.org/T167897) (owner: 10MarcoAurelio) [23:37:34] (03PS2) 10Legoktm: Allow Johanna Mayr (JoHammer) to run the test pipeline [integration/config] - 10https://gerrit.wikimedia.org/r/371757 (owner: 10Catrope) [23:37:36] (03PS4) 10Legoktm: Archive Extension:AdvancedMeta [integration/config] - 10https://gerrit.wikimedia.org/r/371640 (https://phabricator.wikimedia.org/T173184) (owner: 10MarcoAurelio) [23:38:30] (03CR) 10jerkins-bot: [V: 04-1] Archive Extension:ImageTagging [integration/config] - 10https://gerrit.wikimedia.org/r/371653 (https://phabricator.wikimedia.org/T167897) (owner: 10MarcoAurelio) [23:38:47] (03PS2) 10Legoktm: Add config for labs/libraryupgrader [integration/config] - 10https://gerrit.wikimedia.org/r/372208 [23:38:49] (03PS5) 10Legoktm: Archive Extension:ImageTagging [integration/config] - 10https://gerrit.wikimedia.org/r/371653 (https://phabricator.wikimedia.org/T167897) (owner: 10MarcoAurelio) [23:38:51] (03PS3) 10Legoktm: Allow Johanna Mayr (JoHammer) to run the test pipeline [integration/config] - 10https://gerrit.wikimedia.org/r/371757 (owner: 10Catrope) [23:38:53] (03PS5) 10Legoktm: Archive Extension:AdvancedMeta [integration/config] - 10https://gerrit.wikimedia.org/r/371640 (https://phabricator.wikimedia.org/T173184) (owner: 10MarcoAurelio) [23:40:08] (03CR) 10Legoktm: [C: 032] Add config for labs/libraryupgrader [integration/config] - 10https://gerrit.wikimedia.org/r/372208 (owner: 10Legoktm) [23:40:14] (03CR) 10Legoktm: [C: 032] Archive Extension:ImageTagging [integration/config] - 10https://gerrit.wikimedia.org/r/371653 (https://phabricator.wikimedia.org/T167897) (owner: 10MarcoAurelio) [23:40:34] (03PS3) 10Legoktm: Add config for labs/libraryupgrader [integration/config] - 10https://gerrit.wikimedia.org/r/372208 [23:40:36] (03PS6) 10Legoktm: Archive Extension:ImageTagging [integration/config] - 10https://gerrit.wikimedia.org/r/371653 (https://phabricator.wikimedia.org/T167897) (owner: 10MarcoAurelio) [23:40:38] (03PS4) 10Legoktm: Allow Johanna Mayr (JoHammer) to run the test pipeline [integration/config] - 10https://gerrit.wikimedia.org/r/371757 (owner: 10Catrope) [23:40:38] sigh [23:40:40] (03PS6) 10Legoktm: Archive Extension:AdvancedMeta [integration/config] - 10https://gerrit.wikimedia.org/r/371640 (https://phabricator.wikimedia.org/T173184) (owner: 10MarcoAurelio) [23:40:42] (03CR) 10jerkins-bot: [V: 04-1] Archive Extension:ImageTagging [integration/config] - 10https://gerrit.wikimedia.org/r/371653 (https://phabricator.wikimedia.org/T167897) (owner: 10MarcoAurelio) [23:40:49] (03CR) 10Legoktm: [C: 032] Add config for labs/libraryupgrader [integration/config] - 10https://gerrit.wikimedia.org/r/372208 (owner: 10Legoktm) [23:40:59] (03CR) 10Legoktm: [C: 032] Archive Extension:ImageTagging [integration/config] - 10https://gerrit.wikimedia.org/r/371653 (https://phabricator.wikimedia.org/T167897) (owner: 10MarcoAurelio) [23:41:05] (03CR) 10Legoktm: [C: 032] Allow Johanna Mayr (JoHammer) to run the test pipeline [integration/config] - 10https://gerrit.wikimedia.org/r/371757 (owner: 10Catrope) [23:41:10] (03CR) 10Legoktm: [C: 032] Archive Extension:AdvancedMeta [integration/config] - 10https://gerrit.wikimedia.org/r/371640 (https://phabricator.wikimedia.org/T173184) (owner: 10MarcoAurelio) [23:41:36] I think this should work [23:42:12] (03Merged) 10jenkins-bot: Add config for labs/libraryupgrader [integration/config] - 10https://gerrit.wikimedia.org/r/372208 (owner: 10Legoktm) [23:42:47] (03Merged) 10jenkins-bot: Archive Extension:ImageTagging [integration/config] - 10https://gerrit.wikimedia.org/r/371653 (https://phabricator.wikimedia.org/T167897) (owner: 10MarcoAurelio) [23:43:28] (03Merged) 10jenkins-bot: Allow Johanna Mayr (JoHammer) to run the test pipeline [integration/config] - 10https://gerrit.wikimedia.org/r/371757 (owner: 10Catrope) [23:44:11] (03Merged) 10jenkins-bot: Archive Extension:AdvancedMeta [integration/config] - 10https://gerrit.wikimedia.org/r/371640 (https://phabricator.wikimedia.org/T173184) (owner: 10MarcoAurelio) [23:46:19] !log reloading Zuul to deploy https://gerrit.wikimedia.org/r/372208 https://gerrit.wikimedia.org/r/371653 https://gerrit.wikimedia.org/r/371757 https://gerrit.wikimedia.org/r/371640 [23:46:23] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [23:52:11] 10Release-Engineering-Team (Kanban): wmf.14 Blocker - Post Mortem - Cannot flush pre-lock snapshot because writes are pending - https://phabricator.wikimedia.org/T173477#3529618 (10Jrbranaa) [23:54:03] 10Release-Engineering-Team (Kanban): wmf.14 Blocker - Post Mortem - Cannot flush pre-lock snapshot because writes are pending - https://phabricator.wikimedia.org/T173477#3529634 (10Jrbranaa)