[00:00:04] already logged out and back in without being member , heh [00:00:57] I wonder what the statsd_host is supposed to be [00:01:25] I applied the role... [00:01:33] let me force a puppet run [00:02:10] is the statsd_host set in the Hiera: wiki page? [00:02:17] cool [00:02:27] oh fun! [00:02:42] ould not find class role::logstash [00:03:01] that is what i saw before i said something, yea [00:03:03] because it's gone from the puppet master I it's not showing in horizon's ui [00:03:10] oh [00:03:18] this is abug! [00:03:32] :p [00:03:52] is there a way to configure it via commandline? [00:03:55] and it's andrewbogott's bug. I'll write it up [00:04:01] thanks [00:04:14] not that I know of.. [00:04:15] cli that is [00:04:15] i double checked that stuff on all prod logstash [00:04:39] yeah this bug is in the horizon ui that shows what is applied on the host [00:04:50] yep [00:05:00] Project beta-update-databases-eqiad build #13309: 15ABORTED in 45 min: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/13309/ [00:08:00] hey there, I have questions about Selenium/Node.js ... zeljkof you there ? [00:08:20] 10Beta-Cluster-Infrastructure, 10Horizon, 06Labs: Can't remove role::logstash from deployment-logstash2 because the class has been removed from ops/puppet.git - https://phabricator.wikimedia.org/T152472#2849441 (10bd808) [00:08:59] mutante: hopefully andrewbogott and I can figure out a fix for that ^ [00:09:15] but for now puppet being busted on that host isn't the end of the world [00:09:57] this is going to happen with other roles in the prod puppet stuff too so best to figure out the real fix now [00:17:06] Yippee, build fixed! [00:17:07] Project selenium-Flow » chrome,beta,Linux,contintLabsSlave && UbuntuTrusty build #229: 09FIXED in 1 min 5 sec: https://integration.wikimedia.org/ci/job/selenium-Flow/BROWSER=chrome,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=contintLabsSlave%20&&%20UbuntuTrusty/229/ [00:29:49] 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 10Reading-Web-Trending-Service, and 2 others: Setup trending service CI - https://phabricator.wikimedia.org/T151469#2849484 (10Pchelolo) > Change-propagation at the hearth of the system? Trending service... [00:38:56] bd808: ok, thanks! [00:43:38] 10Beta-Cluster-Infrastructure, 10Horizon, 06Labs: Can't remove role::logstash from deployment-logstash2 because the class has been removed from ops/puppet.git - https://phabricator.wikimedia.org/T152472#2849441 (10scfc) I think the problem is caused by the old Puppet configuration in LDAP: ``` scfc@tools-ba... [00:48:25] 10Beta-Cluster-Infrastructure, 10Horizon, 06Labs: Can't remove role::logstash from deployment-logstash2 because the class has been removed from ops/puppet.git - https://phabricator.wikimedia.org/T152472#2849515 (10bd808) I //think// that the Puppet side has been changed to ignore the old LDAP data now. My h... [00:58:19] 03Scap3: scap sync-l10n AttributeError: 'Namespace' object has no attribute 'message' - https://phabricator.wikimedia.org/T152390#2849529 (10thcipriani) [00:58:39] 03Scap3: ElectronPdfService mw extension l10n messages missing after full scap sync - https://phabricator.wikimedia.org/T152424#2849530 (10bd808) * `scap sync-l10n` is not for rebuilding the localization files. It's actually a command that is designed to be used by `l10nupdate` to ship built l10n data to the clu... [01:01:20] 06Release-Engineering-Team, 10Wikimedia-Developer-Summit, 06Developer-Relations (Oct-Dec-2016), 07Documentation: Developer Summit 2017: Work with TPG and RelEng on solution to event documenting - https://phabricator.wikimedia.org/T132400#2849534 (10srishakatux) [01:01:48] 06Release-Engineering-Team, 10Wikimedia-Developer-Summit, 06Developer-Relations (Oct-Dec-2016), 07Documentation: Developer Summit 2017: Work with TPG and RelEng on solution to event documenting - https://phabricator.wikimedia.org/T132400#2196957 (10srishakatux) [01:05:00] Project beta-update-databases-eqiad build #13310: 15ABORTED in 45 min: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/13310/ [02:05:00] Project beta-update-databases-eqiad build #13311: 15ABORTED in 45 min: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/13311/ [03:05:00] Project beta-update-databases-eqiad build #13312: 15ABORTED in 45 min: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/13312/ [03:10:27] 02:20:22 ...Update 'FlowFixLog:version2' already loBuild timed out (after 45 minutes). Marking the build as aborted. [03:10:35] ^ So...that update sucks [03:10:47] Or should be run async from the update.php process... [04:05:00] Project beta-update-databases-eqiad build #13313: 15ABORTED in 45 min: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/13313/ [04:22:31] Or rather, the update after it sucks [04:27:03] 06Release-Engineering-Team, 06Collaboration-Team-Triage, 10Flow, 07Beta-Cluster-reproducible: Maintenance script for causing failed DB updates on Beta - https://phabricator.wikimedia.org/T152479#2849716 (10demon) [04:27:21] 06Release-Engineering-Team, 06Collaboration-Team-Triage, 10Flow, 07Beta-Cluster-reproducible: Maintenance script causing failed DB updates on Beta - https://phabricator.wikimedia.org/T152479#2849729 (10demon) [05:05:00] Project beta-update-databases-eqiad build #13314: 15ABORTED in 45 min: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/13314/ [05:16:48] (03PS1) 10Phuedx: [Popups] depends on EventLogging [integration/config] - 10https://gerrit.wikimedia.org/r/325513 [06:05:00] Project beta-update-databases-eqiad build #13315: 15ABORTED in 45 min: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/13315/ [06:35:36] Yippee, build fixed! [06:35:37] Project selenium-Wikibase » chrome,test,Linux,contintLabsSlave && UbuntuTrusty build #199: 09FIXED in 1 hr 55 min: https://integration.wikimedia.org/ci/job/selenium-Wikibase/BROWSER=chrome,MEDIAWIKI_ENVIRONMENT=test,PLATFORM=Linux,label=contintLabsSlave%20&&%20UbuntuTrusty/199/ [06:52:18] Yippee, build fixed! [06:52:18] Project selenium-Wikibase » chrome,beta,Linux,contintLabsSlave && UbuntuTrusty build #199: 09FIXED in 2 hr 12 min: https://integration.wikimedia.org/ci/job/selenium-Wikibase/BROWSER=chrome,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=contintLabsSlave%20&&%20UbuntuTrusty/199/ [07:05:00] Project beta-update-databases-eqiad build #13316: 15ABORTED in 45 min: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/13316/ [07:24:07] 06Release-Engineering-Team, 06Collaboration-Team-Triage, 10Flow, 07Beta-Cluster-reproducible: Maintenance script causing failed DB updates on Beta - https://phabricator.wikimedia.org/T152479#2849716 (10Catrope) This is a semi-duplicate of {T152379}, except that bug describes an earlier issue where Flow was... [07:48:21] 06Release-Engineering-Team, 10ChangeProp, 06Operations, 06Parsing-Team, and 5 others: Separate clusters for asynchronous processing from the ones for public consumption - https://phabricator.wikimedia.org/T152074#2849909 (10Joe) >>! In T152074#2848311, @GWicke wrote: > There are pros & cons for dividing th... [07:53:01] 06Release-Engineering-Team, 10ChangeProp, 06Operations, 06Parsing-Team, and 5 others: Separate clusters for asynchronous processing from the ones for public consumption - https://phabricator.wikimedia.org/T152074#2849912 (10Joe) So, assuming parsoid can do TLS to its backend (I'll check that), my proposed... [07:54:40] 06Release-Engineering-Team, 10ChangeProp, 06Operations, 06Parsing-Team, and 6 others: Separate clusters for asynchronous processing from the ones for public consumption - https://phabricator.wikimedia.org/T152074#2849914 (10Joe) [08:05:00] Project beta-update-databases-eqiad build #13317: 15ABORTED in 45 min: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/13317/ [08:30:08] 06Release-Engineering-Team, 06Collaboration-Team-Triage, 10Flow, 07Beta-Cluster-reproducible: Beta update.php fails:The content model 'CONTENT_MODEL_FLOW_BOARD' is not registered on this wiki. - https://phabricator.wikimedia.org/T152379#2849944 (10Paladox) This problem could have been caused by extension.j... [08:38:01] 06Release-Engineering-Team, 06Collaboration-Team-Triage, 10Flow, 07Beta-Cluster-reproducible: Beta update.php fails:The content model 'CONTENT_MODEL_FLOW_BOARD' is not registered on this wiki. - https://phabricator.wikimedia.org/T152379#2849946 (10Reedy) >>! In T152379#2849944, @Paladox wrote: > This probl... [08:52:06] 06Release-Engineering-Team, 06Collaboration-Team-Triage, 10Flow, 07Beta-Cluster-reproducible: Beta update.php fails:The content model 'CONTENT_MODEL_FLOW_BOARD' is not registered on this wiki. - https://phabricator.wikimedia.org/T152379#2849955 (10Paladox) Are these configs meant to be in the config sectio... [08:58:28] 06Release-Engineering-Team, 06Collaboration-Team-Triage, 10Flow, 07Beta-Cluster-reproducible: Beta update.php fails:The content model 'CONTENT_MODEL_FLOW_BOARD' is not registered on this wiki. - https://phabricator.wikimedia.org/T152379#2849962 (10Reedy) >>! In T152379#2849955, @Paladox wrote: > Are these... [09:03:13] 06Release-Engineering-Team, 06Collaboration-Team-Triage, 10Flow, 07Beta-Cluster-reproducible: Maintenance script causing failed DB updates on Beta - https://phabricator.wikimedia.org/T152479#2849966 (10Paladox) It seems to now be aborting on 07:20:19 ...rev_user_ip in table flow_revision already modified... [09:05:00] Project beta-update-databases-eqiad build #13318: 15ABORTED in 45 min: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/13318/ [09:07:44] 06Release-Engineering-Team, 06Collaboration-Team-Triage, 10Flow, 07Beta-Cluster-reproducible: Beta update.php fails:The content model 'CONTENT_MODEL_FLOW_BOARD' is not registered on this wiki. - https://phabricator.wikimedia.org/T152379#2849967 (10Paladox) It looks like that config isent in there instead i... [09:07:49] Set $wgShowDBErrorBacktrace = true; at the bottom of LocalSettings.php to show detailed debugging informationcommonswiki: Modifying ref_target field of table flow_ext_ref ...----------------------------------------------------------------- [09:08:12] !log running foreachwiki update.php on beta [09:08:15] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [09:08:21] I think it's timing out because there's a fuckload to do [09:08:57] seems to be a lot of unapplied sql schema changes [09:11:12] 06Release-Engineering-Team, 06Collaboration-Team-Triage, 10Flow, 07Beta-Cluster-reproducible: Maintenance script causing failed DB updates on Beta - https://phabricator.wikimedia.org/T152479#2849987 (10Paladox) It last passed on https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/13275/ so... [09:13:45] 06Release-Engineering-Team, 06Collaboration-Team-Triage, 10Flow, 07Beta-Cluster-reproducible: Maintenance script causing failed DB updates on Beta - https://phabricator.wikimedia.org/T152479#2849716 (10Reedy) The problem seems to be many unapplied schema patches, and jobs that take a long time I'm running... [09:17:25] noisy logs are nosiy [09:19:06] 06Release-Engineering-Team, 06Collaboration-Team-Triage, 10Flow, 07Beta-Cluster-reproducible: Maintenance script causing failed DB updates on Beta - https://phabricator.wikimedia.org/T152479#2849991 (10Paladox) I guess you've found the problem. Probably need to add all those sql files to the updated hook s... [09:24:35] 06Release-Engineering-Team, 06Collaboration-Team-Triage, 10Flow, 07Beta-Cluster-reproducible: Beta update.php fails:The content model 'CONTENT_MODEL_FLOW_BOARD' is not registered on this wiki. - https://phabricator.wikimedia.org/T152379#2849993 (10Reedy) Lots of these on enwiki... ``` enwiki: Checked a t... [10:05:00] Project beta-update-databases-eqiad build #13319: 15ABORTED in 45 min: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/13319/ [10:07:15] 03Scap3: ElectronPdfService mw extension l10n messages missing after full scap sync - https://phabricator.wikimedia.org/T152424#2850045 (10Addshore) >>! In T152424#2849401, @thcipriani wrote: > Was there anything in the output of `scap sync` that looked unusual? Nothing looked odd [11:05:01] Project beta-update-databases-eqiad build #13320: 15ABORTED in 45 min: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/13320/ [11:14:06] 05Continuous-Integration-Scaling: Provision CI:admins ssh public key in Nodepool instances - https://phabricator.wikimedia.org/T128364#2850113 (10hashar) > The LDAP bindpw for the proxyagent user? It's public and shared with all labs instances, so... My understanding was that it is more or less private to labs,... [11:15:14] 05Continuous-Integration-Scaling: Provision CI:admins ssh public key in Nodepool instances - https://phabricator.wikimedia.org/T128364#2850114 (10hashar) There is a very annoying way to get root access on an instance. That requires generating a ssh key pair and spawn an instance with that key pair attached to it... [11:35:47] 10Continuous-Integration-Infrastructure, 05Continuous-Integration-Scaling, 13Patch-For-Review: Speed up the time to get a Nodepool instances to achieve READY state - https://phabricator.wikimedia.org/T113342#2850156 (10hashar) system log for an instance that booted at `Dec 06 10:51:00` (according to `uptime... [11:46:20] 10Continuous-Integration-Infrastructure, 05Continuous-Integration-Scaling, 13Patch-For-Review: Speed up the time to get a Nodepool instances to achieve READY state - https://phabricator.wikimedia.org/T113342#1662355 (10hashar) ``` systemd-analyze blame|head -n10 19.535s networking.service... [11:49:00] hashar, paladox: any change i can get a review of https://gerrit.wikimedia.org/r/#/c/325513/? [11:49:14] sure thing [11:49:35] poor Popups [11:50:14] (03CR) 10Hashar: [C: 032] [Popups] depends on EventLogging [integration/config] - 10https://gerrit.wikimedia.org/r/325513 (owner: 10Phuedx) [11:50:20] all that mess need a better system really :D [11:50:26] but that change is good enough for now ! [11:50:50] phuedx: deploying it in a few [11:51:02] ❤️❤️❤️ [11:51:15] (03Merged) 10jenkins-bot: [Popups] depends on EventLogging [integration/config] - 10https://gerrit.wikimedia.org/r/325513 (owner: 10Phuedx) [12:05:00] Project beta-update-databases-eqiad build #13321: 15ABORTED in 45 min: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/13321/ [12:14:22] 06Release-Engineering-Team, 06Collaboration-Team-Triage, 10Flow, 07Beta-Cluster-reproducible, 13Patch-For-Review: Beta update.php fails:The content model 'CONTENT_MODEL_FLOW_BOARD' is not registered on this wiki. - https://phabricator.wikimedia.org/T152379#2850208 (10SBisson) This is caused by https://ge... [12:22:02] phuedx: done :] [12:22:07] a while ago actually [12:22:14] thanks hashar! [12:23:00] lunch && [12:28:27] (03PS1) 10Zfilipin: Release patch version 1.7.3 [selenium] - 10https://gerrit.wikimedia.org/r/325544 (https://phabricator.wikimedia.org/T152422) [12:31:42] (03CR) 10Zfilipin: [C: 032] Release patch version 1.7.3 [selenium] - 10https://gerrit.wikimedia.org/r/325544 (https://phabricator.wikimedia.org/T152422) (owner: 10Zfilipin) [12:33:57] (03Merged) 10jenkins-bot: Release patch version 1.7.3 [selenium] - 10https://gerrit.wikimedia.org/r/325544 (https://phabricator.wikimedia.org/T152422) (owner: 10Zfilipin) [12:34:31] 10Continuous-Integration-Config, 10Wikidata, 03WMDE-QWERTY-Team-Board: E-Mail notification on failures of Wikidata-builds - https://phabricator.wikimedia.org/T152495#2850267 (10Tobi_WMDE_SW) [12:41:06] 10Browser-Tests-Infrastructure, 13Patch-For-Review, 15User-zeljkofilipin: Release new version of mediawiki_selenium - https://phabricator.wikimedia.org/T152422#2850284 (10zeljkofilipin) 05Open>03Resolved [13:04:29] 10Browser-Tests-Infrastructure, 06Release-Engineering-Team, 15User-zeljkofilipin: Make it possible to execute tests as a specific (new) MediaWiki user on beta cluster - https://phabricator.wikimedia.org/T152432#2850307 (10zeljkofilipin) [13:05:00] Project beta-update-databases-eqiad build #13322: 15ABORTED in 45 min: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/13322/ [13:05:37] 10Browser-Tests-Infrastructure, 15User-zeljkofilipin: Make it possible to execute tests as a specific (new) MediaWiki user on beta cluster - https://phabricator.wikimedia.org/T152432#2848018 (10zeljkofilipin) [13:06:28] 10Browser-Tests-Infrastructure, 15User-zeljkofilipin: Make it possible to execute tests as a specific (new) MediaWiki user on beta cluster - https://phabricator.wikimedia.org/T152432#2848018 (10zeljkofilipin) a:03zeljkofilipin [13:06:45] 10Browser-Tests-Infrastructure, 15User-zeljkofilipin: Make it possible to execute tests as a specific (new) MediaWiki user on beta cluster - https://phabricator.wikimedia.org/T152432#2850314 (10zeljkofilipin) p:05Triage>03Normal [13:21:04] 305 ? Ss 0:00 dhclient -v -pf /run/dhclient.eth0.pid -lf /var/lib/dhcp/dhclient.eth0.leases eth0 [13:21:04] 334 ? Ss 0:00 dhclient -v -pf /run/dhclient.eth0.pid -lf /var/lib/dhcp/dhclient.eth0.leases eth0 [13:21:09] duplicate dhcp clients :( [13:21:10] bah [13:21:26] Yippee, build fixed! [13:21:27] Project beta-update-databases-eqiad build #13323: 09FIXED in 1 min 26 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/13323/ [13:21:46] 06Release-Engineering-Team, 06Collaboration-Team-Triage, 10Flow, 07Beta-Cluster-reproducible: Maintenance script causing failed DB updates on Beta - https://phabricator.wikimedia.org/T152479#2850319 (10Paladox) It seems to be fixed now https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/1... [13:22:33] 06Release-Engineering-Team, 06Collaboration-Team-Triage, 10Flow, 07Beta-Cluster-reproducible, and 2 others: Beta update.php fails:The content model 'CONTENT_MODEL_FLOW_BOARD' is not registered on this wiki. - https://phabricator.wikimedia.org/T152379#2850323 (10Paladox) It is now fixed, see https://integra... [13:22:51] 06Release-Engineering-Team, 06Collaboration-Team-Triage, 10Flow, 07Beta-Cluster-reproducible: Maintenance script causing failed DB updates on Beta - https://phabricator.wikimedia.org/T152479#2850324 (10Paladox) Fixed with https://gerrit.wikimedia.org/r/#/c/325538/ [13:46:26] Yippee, build fixed! [13:46:26] Project selenium-VisualEditor » firefox,beta,Linux,contintLabsSlave && UbuntuTrusty build #235: 09FIXED in 2 min 25 sec: https://integration.wikimedia.org/ci/job/selenium-VisualEditor/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=contintLabsSlave%20&&%20UbuntuTrusty/235/ [13:59:36] 10Browser-Tests-Infrastructure, 06Reading-Web-Backlog, 07Browser-Tests, 13Patch-For-Review, and 2 others: Add helper to Selenium that allows you to query whether JavaScript module has loaded - https://phabricator.wikimedia.org/T146292#2850466 (10zeljkofilipin) 05Open>03Resolved [14:20:06] (03PS1) 10Hashar: dib: fix dupe sourcing in interfaces [integration/config] - 10https://gerrit.wikimedia.org/r/325561 (https://phabricator.wikimedia.org/T113342) [14:28:07] (03PS1) 10Aude: Update Wikidata branch to wmf/1.29.0-wmf.5 [tools/release] - 10https://gerrit.wikimedia.org/r/325562 [14:42:25] (03PS2) 10Hashar: dib: fix dupe sourcing in interfaces [integration/config] - 10https://gerrit.wikimedia.org/r/325561 (https://phabricator.wikimedia.org/T113342) [15:05:33] PROBLEM - Host deployment-elastic08 is DOWN: CRITICAL - Host Unreachable (10.68.21.29) [15:08:19] (03CR) 10Hashar: "Sent upstream as https://gerrit.wikimedia.org/r/#/c/325561/" [integration/config] - 10https://gerrit.wikimedia.org/r/325561 (https://phabricator.wikimedia.org/T113342) (owner: 10Hashar) [15:19:59] 06Release-Engineering-Team, 10ChangeProp, 06Operations, 06Parsing-Team, and 6 others: Separate clusters for asynchronous processing from the ones for public consumption - https://phabricator.wikimedia.org/T152074#2850671 (10Joe) Turns out parsoid's way of contacting the backends wasn't able to support TLS... [15:39:32] 10Browser-Tests-Infrastructure, 15User-zeljkofilipin: Make it possible to execute tests as a specific (new) MediaWiki user on beta cluster - https://phabricator.wikimedia.org/T152432#2850749 (10zeljkofilipin) Every test should create environment it needs (users, pages...) and not assume they will be available... [15:53:19] 10Browser-Tests-Infrastructure, 15User-zeljkofilipin: Make it possible to execute tests as a specific (new) MediaWiki user on beta cluster - https://phabricator.wikimedia.org/T152432#2850800 (10zeljkofilipin) There is a way to bypass CAPTCHA (from [[ https://phabricator.wikimedia.org/diffusion/EGST/browse/mast... [15:58:33] (03CR) 10Aude: [C: 032] Update Wikidata branch to wmf/1.29.0-wmf.5 [tools/release] - 10https://gerrit.wikimedia.org/r/325562 (owner: 10Aude) [16:00:06] (03Merged) 10jenkins-bot: Update Wikidata branch to wmf/1.29.0-wmf.5 [tools/release] - 10https://gerrit.wikimedia.org/r/325562 (owner: 10Aude) [16:11:26] 06Release-Engineering-Team, 10ChangeProp, 06Operations, 06Parsing-Team, and 6 others: Separate clusters for asynchronous processing from the ones for public consumption - https://phabricator.wikimedia.org/T152074#2837406 (10Ottomata) > Not sure what the kafka replication status to codfw Status is A-ok! ht... [16:33:33] !log Nodepool imported a new Jessie image 'jessie-T113342' with some network configuration hotfix. Will use for debugging. T113342 [16:33:38] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [16:44:32] (03PS1) 10Hashar: dib: ensure /etc/apt/sources.list exists [integration/config] - 10https://gerrit.wikimedia.org/r/325576 [16:46:05] hashar hi, remeber the tool i was telling you on friday, well it worked, it can also work if you give it the jenkins url, so it crawls all tests but that requires you to be an admin or have permission to do that. [16:46:42] anyways i managed to do https://gerrit.wikimedia.org/r/#/c/325064/ [16:46:48] and https://gerrit.wikimedia.org/r/#/c/325109/ [16:46:54] (03PS10) 10Paladox: Create operations-puppet-catalog-compiler job in jjb [integration/config] - 10https://gerrit.wikimedia.org/r/325064 (https://phabricator.wikimedia.org/T97513) [16:46:58] (03PS9) 10Paladox: Add a puppet compiler pipeline [integration/config] - 10https://gerrit.wikimedia.org/r/325109 [16:47:55] 10Continuous-Integration-Infrastructure, 05Continuous-Integration-Scaling, 13Patch-For-Review: Speed up the time to get a Nodepool instances to achieve READY state - https://phabricator.wikimedia.org/T113342#2850982 (10hashar) Tried with the new image (that has https://gerrit.wikimedia.org/r/325561) and the... [17:25:43] any opposition to me getting someone in ops to merge https://gerrit.wikimedia.org/r/#/c/315711/ ? It installs the php-ast extension to integration-slave-jessie-* [17:28:05] 06Release-Engineering-Team, 10ChangeProp, 06Operations, 06Parsing-Team, and 6 others: Separate clusters for asynchronous processing from the ones for public consumption - https://phabricator.wikimedia.org/T152074#2851079 (10Joe) >>! In T152074#2850862, @Ottomata wrote: >> Not sure what the kafka replicatio... [17:29:42] 06Release-Engineering-Team, 10ChangeProp, 06Operations, 06Parsing-Team, and 6 others: Separate clusters for asynchronous processing from the ones for public consumption - https://phabricator.wikimedia.org/T152074#2851080 (10Ottomata) Yup! [17:42:55] so after 1 day+ of debugging the boot of Jessie images, I found the main cause of slowness/delay [17:43:04] or how to save 30+ seconds on boot with a single fix https://gerrit.wikimedia.org/r/#/c/325561/2/dib/elements/wikimedia/install.d/11-fix-networking-conf ! [17:56:09] awww, nice one hashar [17:56:54] such a pity it took me 7 or 8 hours to figure it out :d [17:57:04] but I learned a lot more about cloud-init/systemd in the process! [17:59:35] dinner! [18:03:25] 10Gerrit, 06Release-Engineering-Team, 06Operations, 10hardware-requests: Requesting 1 spare misc box for Gerrit in codfw - https://phabricator.wikimedia.org/T148187#2851196 (10RobH) The server allocation of WMF6408 was approved by @mark on sub-task T150885, that sub-task also includes the ordering of dual... [18:15:37] twentyafterfour: Wmf.5 cut soon? Waiting to merge after it. :-) [18:16:08] James_F: not me this week unless I'm horribly uninformed [18:16:34] yep, it's chad this week [18:16:34] twentyafterfour: https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20161206T2000 says it's you but it might be wrong? [18:16:58] James_F: team meeting says ostriches ;) [18:17:07] Kk. Will update. [18:19:53] ostriches: (Same question.) [18:21:51] ostriches: are we doing gerrit2001 ? [18:21:56] 10Gerrit, 06Operations, 10Ops-Access-Requests: Root for Mukunda for Gerrit machine(s) - https://phabricator.wikimedia.org/T152236#2842574 (10Florian) Would be great to have one more person able to maintain gerrit, and from what I know from him, I would trust him! (However, I'm not sure, if my vote counts :P) [18:21:59] instead of element name [18:22:22] i think we probably should, like phab [18:28:11] James_F: Um, starting in like 5 mins [18:28:14] Wrapping something up [18:28:18] mutante: Yes, let's [18:28:29] Awesome, thanks. [18:28:57] I got distracted by monospace fonts for my terminal [18:28:58] :p [18:30:16] * James_F laughs. [18:30:16] James_F: https://phabricator.wikimedia.org/F4994214 [18:30:50] ostriches: I use Andale Mono. [18:31:07] ostriches: Whereas some people just want to watch the world burn, it appears. [18:31:55] 10Beta-Cluster-Infrastructure, 10Horizon, 06Labs, 13Patch-For-Review: Can't remove role::logstash from deployment-logstash2 because the class has been removed from ops/puppet.git - https://phabricator.wikimedia.org/T152472#2851317 (10Andrew) a:03Andrew [18:36:08] 10Gerrit, 06Operations, 10Ops-Access-Requests: Root for Mukunda for Gerrit machine(s) - https://phabricator.wikimedia.org/T152236#2842574 (10mmodell) Thanks Florian :) [18:46:20] 06Release-Engineering-Team, 06Collaboration-Team-Triage, 10Flow, 07Beta-Cluster-reproducible, and 2 others: Beta update.php fails:The content model 'CONTENT_MODEL_FLOW_BOARD' is not registered on this wiki. - https://phabricator.wikimedia.org/T152379#2851374 (10Catrope) 05Open>03Resolved a:03Catrope [18:46:34] James_F: Branch'd [18:47:27] ostriches: Ta! [18:47:35] yw [18:48:03] 06Release-Engineering-Team, 06Collaboration-Team-Triage, 10Flow, 07Beta-Cluster-reproducible: Maintenance script causing failed DB updates on Beta - https://phabricator.wikimedia.org/T152479#2851401 (10Catrope) 05Open>03Resolved a:03Catrope [18:48:34] 06Release-Engineering-Team, 06Collaboration-Team-Triage, 10Flow, 07Beta-Cluster-reproducible: Maintenance script causing failed DB updates on Beta - https://phabricator.wikimedia.org/T152479#2849716 (10Catrope) a:05Catrope>03SBisson [18:51:27] 10Continuous-Integration-Config, 06Discovery, 10Wikimedia-Portals: wikimedia/portals repo might be using outdated and or deprecated tests in jenkins - https://phabricator.wikimedia.org/T152351#2851410 (10MarcoAurelio) I've tried to get that done but then my bot stopped working because it doesn't know on whic... [19:10:16] why are we still running scap from /srv/deployment? on beta: `python /srv/deployment/scap/scap/bin/scap sync beta-scap-eqiad ` [19:13:15] we used to deploy scap with scap to test it the tip of the branch [19:13:20] I guess that is still the case [19:15:47] hashar: sorry to pull you on a tangent, but any opposition to me getting https://gerrit.wikimedia.org/r/#/c/315711/ merged? [19:15:57] its an early step needed for getting static analysis into CI [19:49:11] 10Gerrit, 06Release-Engineering-Team, 06Operations: setup/install gerrit2001/WMF6408 - https://phabricator.wikimedia.org/T152525#2851729 (10RobH) [19:49:19] 10Gerrit, 06Release-Engineering-Team, 06Operations: setup/install gerrit2001/WMF6408 - https://phabricator.wikimedia.org/T152525#2851729 (10RobH) [19:49:58] 10Gerrit, 06Release-Engineering-Team, 06Operations: setup/install gerrit2001/WMF6408 - https://phabricator.wikimedia.org/T152525#2851729 (10RobH) [20:35:31] 05Continuous-Integration-Scaling: Provision CI:admins ssh public key in Nodepool instances - https://phabricator.wikimedia.org/T128364#2852120 (10Krenair) >>! In T128364#2850113, @hashar wrote: >> The LDAP bindpw for the proxyagent user? It's public and shared with all labs instances, so... > > My understanding... [20:43:08] Project selenium-Echo » firefox,beta,Linux,contintLabsSlave && UbuntuTrusty build #233: 04FAILURE in 2 min 7 sec: https://integration.wikimedia.org/ci/job/selenium-Echo/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=contintLabsSlave%20&&%20UbuntuTrusty/233/ [20:43:09] Project selenium-Echo » chrome,beta,Linux,contintLabsSlave && UbuntuTrusty build #233: 04FAILURE in 2 min 9 sec: https://integration.wikimedia.org/ci/job/selenium-Echo/BROWSER=chrome,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=contintLabsSlave%20&&%20UbuntuTrusty/233/ [20:48:14] Project beta-scap-eqiad build #132095: 04FAILURE in 6 min 59 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/132095/ [20:48:44] Ah, dangit [20:48:45] That's me [20:48:48] thcipriani: I missed something [20:49:32] heh, I'm so good at code review [20:49:53] https://phabricator.wikimedia.org/D498 [20:50:30] +1% [20:51:18] Please tell me how no linting job caught that. [20:51:56] that is a good question. [20:52:42] Like, stupid typo on my part, sure...but that shouldn't even lint.... [20:56:42] Project beta-scap-eqiad build #132096: 04STILL FAILING in 6 min 34 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/132096/ [20:59:05] Project beta-scap-eqiad build #132097: 04STILL FAILING in 36 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/132097/ [20:59:39] ^ this should go away soon, FYI [21:01:07] It seems nodepool is stopped again [21:01:16] https://integration.wikimedia.org/zuul/ [21:01:25] It's not stopped but overly slow [21:01:41] hashar ^^ [21:04:42] bah [21:04:52] Amir1: paladox: I am looking [21:05:01] hashar thanks :) [21:05:05] the queues at the bottom of the Zuul status pages are pretty filled [21:05:19] Project beta-scap-eqiad build #132098: 04STILL FAILING in 31 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/132098/ [21:05:49] hashar probaly want to setup a daily chron to restart nodepool [21:05:54] with the test pipeline havng 12-20 changes constantly for the last two hours [21:05:58] paladox: na it is irrelevant [21:06:07] Oh [21:06:26] https://grafana.wikimedia.org/dashboard/db/nodepool shows the state of the pool [21:06:52] used seems to be high [21:07:03] yeah lot of changes [21:07:06] where as looking at ready only shows 2 [21:08:12] and the tasks per minutes graph on ( https://grafana.wikimedia.org/dashboard/db/nodepool ) shows that we have reached the max speed to send queries to OpenStack [21:08:23] or 9/10 per minutes [21:08:35] Oh, is there any way to bump that? [21:08:38] so basically [21:08:59] it is throttled [21:09:12] Oh [21:09:21] Is there a way to get it un throttled? [21:09:32] or did openstack do it so it is throttled? [21:10:01] nodepool.provider_manager.DeleteServerTask object at 0x7fc8246ca190> (queue: 3) [21:10:16] that is 3 tasks waiting to delete an instance [21:10:21] oh [21:10:51] it process one task every 6 seconds (or 10 per minute) [21:11:08] so if we gotta delete 5 instances, that takes 5 x 6 seconds = 30 seconds to process [21:11:09] oh, i guess that's why we hit the slowness [21:11:17] yep [21:11:26] so yeah, it does not go fast enough [21:11:57] yep, i wonder did openstack implement support for it to go faster? [21:12:55] so it is working [21:13:08] ok [21:13:09] :) [21:13:39] !log Refresh Nodepool Jessie snapshot which boot 3 times faster. Will help get nodes available faster T113342 [21:13:43] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [21:14:04] hashar http://docs.openstack.org/kilo/config-reference/content/configuring-compute-API.html [21:14:05] that is going to be hit or miss [21:14:06] 10Continuous-Integration-Config: tox-jessie is failing on operations/software - https://phabricator.wikimedia.org/T152549#2852279 (10scfc) [21:14:13] I wanted to deploy that image tomorro [21:14:24] but I guess we will see immediate benefits if that works all ok [21:14:27] http://docs.openstack.org/kilo/config-reference/content/object-storage-rate-limits.html [21:14:31] ok [21:15:24] Project beta-scap-eqiad build #132099: 04STILL FAILING in 36 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/132099/ [21:17:31] 10Continuous-Integration-Config, 06Operations, 06Operations-Software-Development: tox-jessie is failing on operations/software - https://phabricator.wikimedia.org/T152549#2852322 (10hashar) There are a bunch of flake8 errors. Most probably flake8 is not pinned to a specific version and thus use the latest on... [21:19:37] hashar http://lists.openstack.org/pipermail/openstack-infra/2015-March/002490.html [21:19:43] 10Continuous-Integration-Config, 06Operations, 06Operations-Software-Development: tox-jessie is failing on operations/software - https://phabricator.wikimedia.org/T152549#2852326 (10hashar) Looking at previously merged changes, https://gerrit.wikimedia.org/r/#/c/322619/ failed on November 21th. So something... [21:20:15] http://lists.openstack.org/pipermail/openstack-infra/2015-March/002493.html [21:20:30] paladox: yeah you want Jeblair reply http://lists.openstack.org/pipermail/openstack-infra/2015-March/002493.html :D [21:20:31] yeah that one [21:20:43] Oh [21:20:44] yep [21:20:48] !log Image ci-jessie-wikimedia-1481058839 in wmflabs-eqiad is ready T113342 [21:20:52] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [21:20:55] would setting the rate to 0.1 improve things? [21:21:31] yes [21:21:33] that is what we had [21:21:39] oh [21:21:49] basically [21:22:44] 10Continuous-Integration-Config: npm-node-4 job name should end with "-jessie" - https://phabricator.wikimedia.org/T152552#2852338 (10Legoktm) [21:22:59] when Nodepool has to do queries to openstack, it creates an internal task, put in a queue and process with something else [21:23:13] the queue of queries to make to openstack are then processed at a fixed rate [21:23:29] that is really serializaing the requests [21:23:57] when the queue grow faster than the rate can process items... it grows and the system lags [21:24:20] I will spend some time merging the composer-php55 and composer-hhvm jobs today [21:24:44] but that's on trusty, and it looks like jessie is the one that's behind... [21:26:23] (03PS1) 10Paladox: Rename npm-node-4 test to npm-node-4-jessie [integration/config] - 10https://gerrit.wikimedia.org/r/325635 (https://phabricator.wikimedia.org/T152552) [21:26:32] yep [21:27:46] legoktm: o/ This is a patch to fix a UBN! bug. Can you take a look? https://gerrit.wikimedia.org/r/#/c/325624/ [21:27:52] super straightforward [21:28:36] hashar i managed to do https://gerrit.wikimedia.org/r/#/c/325064/ [21:28:37] :) [21:28:47] that tool i was talking about, actually worked [21:28:59] Amir1: looking [21:29:20] it can also query with the jenkins link as long as you are a admin so we can convert the other jobs to jjb. [21:29:32] legoktm: thanks [21:29:35] Amir1: does that need to be backported and deployed? [21:29:45] Yup. I can do it [21:29:52] Just waiting to be merged [21:30:14] ok [21:30:17] Amir1: Thx for the quick fix on that [21:30:56] legoktm: the hhvm jobs should move to Jessie [21:31:07] ostriches: thanks for finding out. Can I deploy now? or we should wait until SWAT? [21:31:21] Go ahead now, when you're ready :) [21:31:48] legoktm: what puzzles me with the composer-hhvm vs composer-php55 is that in lot of cases they have a lot of overlap :( [21:32:03] Thanks! [21:32:03] eg CodeSniffer being run twice [21:32:34] Or we could stop caring about php55 for testing cuz it's slow ;-) [21:32:34] hashar we can move wikimedia/fundraising/dash to node-4.3 [21:32:40] i fixed the package [21:32:47] here https://gerrit.wikimedia.org/r/#/c/277768/ [21:33:03] ostriches: actually php55 runs phplint much faster than hhvm :) [21:33:07] paladox: will want to check with them. IIRC they were still using node 0.10 ? [21:33:12] so we can drop node-0.10 once we fix pywikibot/i18n [21:33:13] legoktm: lint or unit? [21:33:19] hashar: and normally it's the same, but sometimes the two versions act differently -.- https://github.com/squizlabs/PHP_CodeSniffer/pull/669 [21:33:27] ostriches: lint, which is what the composer-php55/hhvm jobs run [21:33:32] Well, linting is dumb to do server side anyway. [21:33:37] why? [21:33:37] hhvm has more startup overhead compared to Zend (parsing/jit etc) [21:33:39] * ostriches mumbles something about .arclint [21:33:41] ;-) [21:33:44] ejegg awight hi, im wondering are you still using node 0.10 for wikimedia/fundraising/dash? [21:33:54] php55 unit already only runs on gate [21:34:19] paladox: sadly, yes [21:34:22] ostriches: it usually catches most GCI students on their first patch :P [21:34:22] * ostriches goes back to doing useful things instead of trolling [21:34:23] Oh [21:34:35] legoktm: So would arc ;P [21:34:47] ejegg i thought i fixed it?, i guess we can add node-4.3. So it tests with 0.10 and 4.3. [21:34:51] we didn't have time to upgrade that box to newer debian before big english infrastructure lockdown [21:35:25] oh [21:35:31] ejegg: iirc some dependency was not compatible for 4.3 and as I understood it we wanted to keep running tests with node0.10 [21:35:38] so I guess we want to keep the status quo [21:35:54] yeah, there was that one syslog package that paladox found a replacement for [21:36:07] yep [21:36:08] https://gerrit.wikimedia.org/r/#/c/277768/3 [21:36:11] but unfortunately it wouldn't run with the ancient node that's in prod [21:36:44] oh, isen't there a way to update prod [21:36:47] Yippee, build fixed! [21:36:48] Project beta-scap-eqiad build #132100: 09FIXED in 11 min: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/132100/ [21:37:01] using the offical source [21:37:02] https://nodejs.org/en/ [21:37:05] ejegg ^^ [21:37:31] https://nodejs.org/en/download/package-manager/#debian-and-ubuntu-based-linux-distributions [21:38:06] paladox: thanks! I'll pass that along to Jeff_Green [21:38:16] Your welcome [21:38:31] curl -sL https://deb.nodesource.com/setup_7.x | sudo -E bash - [21:38:34] Bahahahahahahaha [21:38:39] Like we'd ever run that in prod. [21:38:39] Yep [21:38:43] What a joke. [21:38:44] oh [21:40:21] ostriches it's open sourced the script too [21:40:24] https://github.com/nodesource/distributions/blob/master/deb/setup_7.x [21:40:37] yah, that just adds the repo, right? [21:40:51] that's not the point at all. [21:40:54] ejegg yeh, you have to after doing that do this [21:41:07] sudo apt-get install -y nodejs [21:41:13] piping curl to bash is a fast ticket to heartache [21:41:19] Oh [21:41:31] https://www.idontplaydarts.com/2016/04/detecting-curl-pipe-bash-server-side/ [21:41:32] ejegg also if your using an old distro see https://github.com/nodesource/distributions/blob/master/OLDER_DISTROS.md [21:41:38] hey zeljkof you there? I have some questions about Selenium/Node.js [21:42:03] ostriches you could always run wget instead of curl then run the script like [21:42:10] ./setup_7.x [21:43:05] That's still not the point. We'd backport the package, if anything [21:43:10] And install it from our own apt. [21:43:16] Oh [21:43:29] I guess ostriches idea is the safest [21:43:39] Anyway, I don't like node anyway, I'm mostly trolling [21:44:12] Oh why doint you like node? [21:44:17] that whole setup script just figures out whether your distro is supported, adds the right repo to sources.list.d, and runs apt-get update [21:44:32] yep [21:44:46] because I don't like javascript [21:44:58] wget -qO- https://deb.nodesource.com/setup_4.x | sudo bash - [21:45:04] oh [21:45:04] well, and adds the apt gpg key [21:45:46] so, easy enough to do in puppet w/o running their .sh [21:45:56] Yep [21:48:10] Amir1: Probably want to port to wmf.5 too? Or just 4? [21:48:31] I'll port to wmf.4 deploy and then wmf.5 if that's fine [21:48:41] jenkins is killing me.... [21:48:48] 10Continuous-Integration-Infrastructure, 05Continuous-Integration-Scaling, 13Patch-For-Review: Speed up the time to get a Nodepool instances to achieve READY state - https://phabricator.wikimedia.org/T113342#2852432 (10hashar) Deployment of the new image seems all fine. Can't tell whether it has much impact... [21:48:59] Amir1: Yeah, it's a tad lagged right now. No worries, we'll get it out :) [21:49:27] :) [21:50:45] hashar the best idea for the next update openstack will do with nodepool is to make it like Docker but also allow it to create instances and delete on the fly. [21:51:09] that will make it faster and will allow it to be more peformant and + secure + more resources :) [21:51:36] as it will create mutiple Docker types on the same instance if it has run out of it's capacity. [21:53:10] 10Gerrit, 06Operations, 10Ops-Access-Requests: Root for Mukunda for Gerrit machine(s) - https://phabricator.wikimedia.org/T152236#2852453 (10MarcoAurelio) Same as Florian. Not sure if my opinion counts (left a lovely token above just in case) but I think @mmodell would be a good Gerrit mantainer. [22:00:59] A just crazy idea, I have no idea how CI infra works but why not using k8s? [22:01:55] paladox: Amir1: we dont know yet about the future [22:02:07] but yes we had a quick proof of concept with Docker [22:02:14] ok [22:02:26] then rebuilding a brand new architecture is going to take a while (a few months at least) [22:02:35] and few more months to migrate [22:02:45] hashar what about a donate to releng page so releng get's funding for more resources in ci? [22:02:47] so if we go that road, will probably keep us busy for most of 2017 [22:03:01] CI is funded [22:03:10] it is 100% under WMF umbrella [22:03:20] specially Technology group [22:03:27] i mean resources as in more servers for releng to use for ci [22:03:31] then it is not like the WMF has billions of dollars on the table [22:03:45] to spend on engineers, more servers, a dedicated infra etc [22:03:49] paladox: We don't (generally) accept directed donations. [22:03:53] oh [22:03:58] So you can't donate saying "This money is only good for XYZ" [22:04:08] ok [22:04:14] there are also different priorityies [22:04:17] Sometimes we make exceptions for grants, but they're few and far between these days [22:04:23] They're very constraining [22:04:24] the WMF budget is relatively small compared to other org/companies [22:04:25] is there something specifically that slows it down right now [22:04:30] oh [22:04:33] mutante: Too much testing ;-) [22:04:53] fair [22:05:16] so we could do great things a million dollars or so. Then they are probably better spent on other areas [22:06:31] hmm, currently we got to "people start to do manual V+2" [22:07:53] heh, a donate to releng page, make the individual depts do their own fundraising.... [22:09:32] BUILD FAILED. Your credit is exhausted. [22:09:34] greg-g: Suddenly, half of each team's time will deal with fundraising :) [22:09:39] would be awkward [22:10:02] :) [22:10:23] I make a PAN form for myself to transition to Fundraising Tech :P [22:10:32] speak of the devil [22:13:02] so for the huge backlog in CI tonight [22:13:11] I think the reason is that bunch of changes got CR+2 [22:13:32] thus the gate-and-pipeline has been fairly active for the last few hours [22:13:32] and [22:13:45] since gate-and-pipeline jobs have an higher precedence over verything else [22:14:01] the test pipeline filled up because there were not enough room for them [22:18:18] engouh for today [22:18:21] * hashar waves [22:18:35] * greg-g waves [22:18:55] bye hashar [22:22:52] 10Beta-Cluster-Infrastructure, 10Horizon, 06Labs, 13Patch-For-Review: Can't remove role::logstash from deployment-logstash2 because the class has been removed from ops/puppet.git - https://phabricator.wikimedia.org/T152472#2852578 (10bd808) 05Open>03Resolved ``` deployment-logstash2.deployment-prep:~ b... [22:34:29] RECOVERY - Puppet run on deployment-logstash2 is OK: OK: Less than 1.00% above the threshold [0.0] [22:37:13] ^ nice [23:24:44] 06Release-Engineering-Team (Deployment-Blockers), 05Release: MW-1.29.0-wmf.6 deployment blockers - https://phabricator.wikimedia.org/T152563#2852689 (10greg) [23:24:57] 06Release-Engineering-Team (Deployment-Blockers), 05Release: MW-1.29.0-wmf.5 deployment blockers - https://phabricator.wikimedia.org/T150972#2803096 (10greg) [23:25:19] 06Release-Engineering-Team (Deployment-Blockers), 05Release: MW-1.29.0-wmf.6 deployment blockers - https://phabricator.wikimedia.org/T152563#2852689 (10greg) a:03mmodell [23:26:05] 06Release-Engineering-Team (Deployment-Blockers), 05Release: MW-1.29.0-wmf.5 deployment blockers - https://phabricator.wikimedia.org/T150972#2803096 (10greg) [23:26:27] 06Release-Engineering-Team (Deployment-Blockers), 05Release: MW-1.29.0-wmf.6 deployment blockers - https://phabricator.wikimedia.org/T152563#2852689 (10greg) [23:40:18] 06Release-Engineering-Team, 15User-greg: Pre 2016 offsite "must do"s - https://phabricator.wikimedia.org/T140923#2852840 (10greg) 05Open>03Resolved a:03greg heh, oops, forgot about this one [23:42:12] 06Release-Engineering-Team, 15User-greg: Setup monthly "QA Tribe" meeting - https://phabricator.wikimedia.org/T152570#2852845 (10greg)