[02:36:02] Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/R96340 was modified, changed by R96340 link https://wikitech.wikimedia.org/w/index.php?diff=1758543 edit summary: [02:37:42] Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/R96340 was modified, changed by R96340 link https://wikitech.wikimedia.org/w/index.php?diff=1758544 edit summary: [02:40:22] Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/R96340 was modified, changed by R96340 link https://wikitech.wikimedia.org/w/index.php?diff=1758545 edit summary: [02:42:57] Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/R96340 was modified, changed by R96340 link https://wikitech.wikimedia.org/w/index.php?diff=1758546 edit summary: [03:03:43] 10Tool-Labs-tools-Xtools: Remove test section on xtools-articleinfo - https://phabricator.wikimedia.org/T164683#3241827 (10Matthewrbowker) p:05Triage>03Lowest a:05Cyberpower678>03None If this only shows up in the legacy xtools, we can close this task. @MusikAnimal could you verify please? I'm still set... [03:16:40] 10Tool-Labs-tools-Xtools: Remove test section on xtools-articleinfo - https://phabricator.wikimedia.org/T164683#3242527 (10MusikAnimal) 05Open>03declined Yes, this is only an issue with the legacy articleinfo. [04:43:56] Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/R96340 was modified, changed by R96340 link https://wikitech.wikimedia.org/w/index.php?diff=1758548 edit summary: [06:35:45] PROBLEM - Puppet errors on tools-exec-1414 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [07:15:45] RECOVERY - Puppet errors on tools-exec-1414 is OK: OK: Less than 1.00% above the threshold [0.0] [09:21:24] Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/FriyMan was created, changed by FriyMan link https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Access_Request/FriyMan edit summary: Created page with "{{Tools Access Request |Justification=I would like to learn how the tools works, as well as coming up with ideas for tools, that would ease our work on wikipedia. |Completed=f..." [11:17:59] 06Labs, 10Tool-Labs, 07Tracking: Tools that should get deleted (tracking) - https://phabricator.wikimedia.org/T133777#3243295 (10zhuyifei1999) [11:18:01] 06Labs: Delete tools.datbotcommons - https://phabricator.wikimedia.org/T164662#3243294 (10zhuyifei1999) [12:11:43] !log wikispeech Deploy latest from Git master: 499e9b6, a031903 (T162361), 891162a [12:11:46] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Wikispeech/SAL [12:11:46] T162361: Add mockup for feedback page - https://phabricator.wikimedia.org/T162361 [13:14:41] Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/Stefan Schneider (WMDE) was created, changed by Stefan Schneider (WMDE) link https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Access_Request/Stefan_Schneider_(WMDE) edit summary: Created page with "{{Tools Access Request |Justification=I am employee of WMDE and would like to use the platform to test some opportunities of the tool platform. |Completed=false |User Name=Ste..." [13:15:37] 06Labs, 06Operations: Stretch vs. Salt - https://phabricator.wikimedia.org/T164595#3239263 (10faidon) Why are you trying to install the jessie version to stretch boxes? In the test box that we've had in prod, the stretch version of salt (as shipped by Debian) works fine with our jessie master. There were some... [13:26:39] quick question, do ssh keys added via gerrit web ui sync with wikitech? [13:27:05] Zppix: not to my knowledge no [13:27:33] ok :/ [13:43:46] 06Labs, 10Labs-Infrastructure: labservices1002 slow puppet runs - https://phabricator.wikimedia.org/T164675#3241620 (10chasemp) labservices1002 ```Notice: Finished catalog run in 346.38 seconds Debug: Executing '/etc/puppet/etckeeper-commit-post' Debug: Closing connection for https://puppet:8140 Debug: Creati... [13:47:32] 06Labs, 06Operations: Stretch vs. Salt - https://phabricator.wikimedia.org/T164595#3243832 (10Andrew) 05Open>03Invalid There was some discussion on IRC about this, during which we were briefly in agreement that we should be running the same minion on all hosts. That's clearly not going to happen so I've m... [13:47:42] 06Labs, 06Operations: Stretch vs. Salt - https://phabricator.wikimedia.org/T164595#3243834 (10akosiaris) >>! In T164595#3243687, @faidon wrote: > Why are you trying to install the jessie version to stretch boxes? In the test box that we've had in prod, the stretch version of salt (as shipped by Debian) works f... [13:50:58] 06Labs, 10Labs-Infrastructure: labservices1002 slow puppet runs - https://phabricator.wikimedia.org/T164675#3243848 (10chasemp) > /dev/md0 ext3 9.1G 3.0G 5.7G 34% / > tune2fs -l /dev/md0 ```tune2fs 1.42.9 (4-Feb-2014) Filesystem volume name: Last mounted on: / Filesystem UUID:... [13:51:06] 06Labs, 10Labs-Infrastructure: labservices1002 slow puppet runs - https://phabricator.wikimedia.org/T164675#3243849 (10chasemp) > !log labservices1002 'touch /forcefsck && sudo reboot' [13:57:26] 06Labs, 10Labs-Infrastructure: labservices1002 slow puppet runs - https://phabricator.wikimedia.org/T164675#3243895 (10chasemp) ```May 7 17:14:58 labservices1002 kernel: [ 15.530367] EXT4-fs (md0): mounting ext3 file system using the ext4 subsystem May 7 17:14:58 labservices1002 kernel: [ 15.540377] EXT4... [14:08:35] 06Labs, 10Labs-Infrastructure: labservices1002 slow puppet runs - https://phabricator.wikimedia.org/T164675#3243962 (10chasemp) > tune2fs -l /dev/md0 | grep 'Last checked' Last checked: Mon May 8 13:50:42 2017 [14:29:00] 06Labs, 10Labs-Infrastructure: labservices1002 slow puppet runs - https://phabricator.wikimedia.org/T164675#3244077 (10chasemp) fsck seems to run fine but is later in the boot process than this which shows up every boot ```[ 15.165191] md0: detected capacity change from 0 to 9990701056 [ 15.204433] md0:... [14:38:58] 06Labs, 10Labs-Infrastructure: labservices1002 slow puppet runs - https://phabricator.wikimedia.org/T164675#3244143 (10chasemp) > time aptitude update ```Ign http://mirrors.wikimedia.org trusty InRelease Hit http://apt.wikimedia.org trusty-wikimedia InRelease Hit http://mirrors.wikimedia.org trusty-updates In... [14:53:54] 06Labs, 10Analytics, 10Pageviews-API, 10wikitech.wikimedia.org: wikitech.wikimedia.org missing from pageviews API - https://phabricator.wikimedia.org/T153821#3244188 (10Milimetric) p:05Triage>03High [15:08:25] 06Labs, 10Labs-Infrastructure: labservices1002 slow puppet runs and IO issues - https://phabricator.wikimedia.org/T164675#3244244 (10chasemp) [15:13:27] 06Labs, 06DC-Ops, 06Operations: Move labstore1002 and labstore1002-array1 and labstore1002-array2 to different rack (currently in C3) - https://phabricator.wikimedia.org/T158913#3244260 (10Cmjohnson) I really do not know where else they can go...each one of these requires 6u of space and has to be in (prefer... [15:15:15] 06Labs, 10Labs-Infrastructure: labservices1002 slow puppet runs and IO issues - https://phabricator.wikimedia.org/T164675#3244265 (10chasemp) I tried dropping into LifeCycle controller to run diagnostics via F-10 https://www.dell.com/support/article/us/en/19/SLN283546/how-to-run-hardware-diagnostics-on-your-p... [15:19:15] 06Labs, 10Analytics, 10Pageviews-API, 10wikitech.wikimedia.org: wikitech.wikimedia.org missing from pageviews API - https://phabricator.wikimedia.org/T153821#2892095 (10bd808) >>! In T153821#2965325, @Milimetric wrote: > Are there plans on putting wikitech behind varnish? Indirectly, yes via {T161859}. At... [15:20:20] Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/R96340 was modified, changed by BryanDavis link https://wikitech.wikimedia.org/w/index.php?diff=1758605 edit summary: [15:22:42] 06Labs, 10Analytics, 10Pageviews-API, 10wikitech.wikimedia.org: wikitech.wikimedia.org missing from pageviews API - https://phabricator.wikimedia.org/T153821#3244283 (10bd808) p:05High>03Normal Lowering priority from high to normal. Having pageview data on wikitech would be nice, but I don't see that i... [15:23:30] 06Labs, 10Labs-Infrastructure: Try to make a Stretch base image for Labs - https://phabricator.wikimedia.org/T164273#3244287 (10Andrew) Stretch images can now be built on labs-bootstrapvz-stretch.openstack.eqiad.wmflabs. A few Ops are working with test builds now... if everyone is happy then we can consider m... [15:27:08] 10Tool-Labs-tools-Xtools, 06Community-Tech: Epic: Rewriting XTools - https://phabricator.wikimedia.org/T153112#3244309 (10Matthewrbowker) [15:40:09] Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/Stefan Schneider (WMDE) was modified, changed by BryanDavis link https://wikitech.wikimedia.org/w/index.php?diff=1758608 edit summary: [15:44:58] Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/FriyMan was modified, changed by BryanDavis link https://wikitech.wikimedia.org/w/index.php?diff=1758611 edit summary: [15:46:46] Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/Cabayi was modified, changed by BryanDavis link https://wikitech.wikimedia.org/w/index.php?diff=1758613 edit summary: [15:48:47] Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/Rockroland was modified, changed by BryanDavis link https://wikitech.wikimedia.org/w/index.php?diff=1758615 edit summary: [15:50:18] Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/Xephyr826 was modified, changed by BryanDavis link https://wikitech.wikimedia.org/w/index.php?diff=1758617 edit summary: [15:53:36] Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/Psubhashish was modified, changed by BryanDavis link https://wikitech.wikimedia.org/w/index.php?diff=1758619 edit summary: [15:58:36] Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/Kushim was modified, changed by BryanDavis link https://wikitech.wikimedia.org/w/index.php?diff=1758621 edit summary: [15:59:11] Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/Psubhashish was modified, changed by BryanDavis link https://wikitech.wikimedia.org/w/index.php?diff=1758622 edit summary: [16:09:27] 06Labs, 10Labs-Infrastructure: labservices1002 slow puppet runs and IO issues - https://phabricator.wikimedia.org/T164675#3244459 (10chasemp) I updated `/etc/default/rcS`: ```# be more verbose during the boot process VERBOSE=yes # automatically repair filesystems with inconsistencies during boot FSCKFIX=yes`... [16:11:54] Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/Cookies52 was modified, changed by BryanDavis link https://wikitech.wikimedia.org/w/index.php?diff=1758625 edit summary: Add note to requesting user about 2FA reset [16:14:47] 10Tool-Labs-tools-Xtools, 03Community-Tech-Sprint, 07I18n: XTools: Cleanup i18n messages - https://phabricator.wikimedia.org/T163468#3244465 (10MusikAnimal) Relevant commit, though I don't recommend trying to review it (101 files :) https://github.com/x-tools/xtools-rebirth/commit/fcf66fab76752b49079ca240870... [16:28:56] Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/MArostegui was modified, changed by BryanDavis link https://wikitech.wikimedia.org/w/index.php?diff=1758628 edit summary: [16:29:36] 06Labs, 10Labs-Infrastructure: labservices1002 slow puppet runs and IO issues - https://phabricator.wikimedia.org/T164675#3244580 (10chasemp) 05Open>03stalled Well. Now the issue has gone dormant or fsck addressed it. ```rush@labservices1002:~# time sudo puppet agent --test Info: Retrieving plugin Info: L... [16:30:04] 06Labs, 10Labs-Infrastructure: labservices1002 slow puppet runs and IO issues - https://phabricator.wikimedia.org/T164675#3244585 (10chasemp) p:05Triage>03Normal [16:30:42] 06Labs, 10Labs-Infrastructure: labservices1002 slow puppet runs and IO issues - https://phabricator.wikimedia.org/T164675#3244591 (10Andrew) a:03Andrew [16:31:15] Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/Acagastya was modified, changed by BryanDavis link https://wikitech.wikimedia.org/w/index.php?diff=1758630 edit summary: [16:41:41] 06Labs, 06DC-Ops, 06Operations: Move labstore1002 and labstore1002-array1 and labstore1002-array2 to different rack (currently in C3) - https://phabricator.wikimedia.org/T158913#3244646 (10madhuvishy) @Cmjohnson and I spoke a bit on IRC, and we talked about moving back to row C, and possibly to C2 and C3, pr... [16:48:05] 06Labs, 06DC-Ops, 06Operations, 10ops-eqiad: Move labstore1002 and labstore1002-array1 and labstore1002-array2 to different rack (currently in C3) - https://phabricator.wikimedia.org/T158913#3244672 (10Cmjohnson) [17:07:31] 06Labs, 06DC-Ops, 06Operations, 10ops-eqiad: Move labstore1002 and labstore1002-array1 and labstore1002-array2 to different rack (currently in C3) - https://phabricator.wikimedia.org/T158913#3051441 (10jcrespo) No problem with me on this, it just takes a few ( minutes to depool/shutdown mysql/shutdown, etc... [17:07:57] 06Labs, 10Labs-Infrastructure: labservices1002 slow puppet runs and IO issues - https://phabricator.wikimedia.org/T164675#3241620 (10faidon) I can't imagine how an fsck would fix a slow puppet run. labservices1002 seems to be an R320, are you sure you weren't affected by T162850? There is a puppet fix committe... [17:08:14] 06Labs, 06DC-Ops, 06Operations, 10ops-eqiad: Move labstore1002 and labstore1002-array1 and labstore1002-array2 to different rack (currently in C3) - https://phabricator.wikimedia.org/T158913#3244795 (10jcrespo) CC @marostegui, in case I was away on vacation. [17:36:28] 06Labs, 10Labs-Infrastructure: Audit disk usage on labvirts - https://phabricator.wikimedia.org/T163796#3244882 (10Andrew) p:05High>03Normal [17:48:30] 06Labs, 10Labs-Infrastructure: labservices1002 slow puppet runs and IO issues - https://phabricator.wikimedia.org/T164675#3244900 (10chasemp) I'm not sure what the story is other than during debugging it disappeared for now. Pattern as I understand it is: puppet slow seemingly from degraded IO sunday, failed... [17:53:28] (03CR) 10BryanDavis: [C: 032] Implement Tool Labs membership application and processing [labs/striker/deploy] - 10https://gerrit.wikimedia.org/r/351175 (https://phabricator.wikimedia.org/T162508) (owner: 10BryanDavis) [17:53:34] (03Merged) 10jenkins-bot: Implement Tool Labs membership application and processing [labs/striker/deploy] - 10https://gerrit.wikimedia.org/r/351175 (https://phabricator.wikimedia.org/T162508) (owner: 10BryanDavis) [18:16:16] 06Labs, 10Labs-Infrastructure: Try to make a Stretch base image for Labs - https://phabricator.wikimedia.org/T164273#3245023 (10Andrew) 05Open>03Resolved I updated the docs about this, at least slightly. https://wikitech.wikimedia.org/wiki/OpenStack#Building_new_images should be current and fairly accurate. [18:35:06] PROBLEM - Puppet errors on tools-exec-1440 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [19:08:18] 06Labs, 06Operations: Stretch vs. Salt - https://phabricator.wikimedia.org/T164595#3245223 (10faidon) We should stop relying on (ancient versions of) SaltStack altogether, for security reasons among other things. I personally prefer the risk of wildly different salt versions compared to wildly outdated salt ve... [19:10:03] RECOVERY - Puppet errors on tools-exec-1440 is OK: OK: Less than 1.00% above the threshold [0.0] [19:16:00] (03PS1) 10Andrew Bogott: Added dummy uwsgi creds [labs/private] - 10https://gerrit.wikimedia.org/r/352644 [19:16:17] (03PS2) 10Andrew Bogott: Added dummy striker creds [labs/private] - 10https://gerrit.wikimedia.org/r/352644 [19:17:43] 06Labs, 10Labs-Infrastructure: labservices1002 slow puppet runs and IO issues - https://phabricator.wikimedia.org/T164675#3245264 (10faidon) I audited w/ @Dzahn the Icinga check that we have. It's not broken, at least with recent kernels, but it wouldn't fire off because the `cpufreq` module isn't present/load... [19:21:58] 06Labs, 10Labs-Infrastructure: labservices1002 slow puppet runs and IO issues - https://phabricator.wikimedia.org/T164675#3245299 (10Andrew) That system is safe to reboot right now... any pointers about where I should look in the bios for that setting? (I can dig if not) [19:32:46] 06Labs, 10Labs-Infrastructure: labservices1002 slow puppet runs and IO issues - https://phabricator.wikimedia.org/T164675#3245364 (10faidon) Should be System Profile Settings > System Profile -> Performance Per Watt (OS), according to [[ https://wikitech.wikimedia.org/wiki/Platform-specific_documentation/Dell_... [19:37:52] 06Labs, 10Striker, 10Tool-Labs, 15User-bd808: Keystone permissions exception when trying to approve Tool Labs membership request via Striker - https://phabricator.wikimedia.org/T164787#3245386 (10bd808) [20:02:11] 06Labs, 10Striker, 10Tool-Labs, 15User-bd808: Keystone permissions exception when trying to approve Tool Labs membership request via Striker - https://phabricator.wikimedia.org/T164787#3245482 (10bd808) Granting the role using the `openstack` cli tool from californium works as expected: ``` $ openstack rol... [20:19:07] 06Labs, 10Striker, 10Tool-Labs, 15User-bd808: Keystone permissions exception when trying to approve Tool Labs membership request via Striker - https://phabricator.wikimedia.org/T164787#3245534 (10bd808) My current hunch is that this is being blocked because Striker is asking to talk to the `public` endpoin... [20:24:42] PROBLEM - Puppet errors on tools-exec-1437 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [20:44:39] (03PS3) 10BryanDavis: Added dummy striker creds [labs/private] - 10https://gerrit.wikimedia.org/r/352644 (owner: 10Andrew Bogott) [20:44:49] (03CR) 10BryanDavis: [C: 031] Added dummy striker creds [labs/private] - 10https://gerrit.wikimedia.org/r/352644 (owner: 10Andrew Bogott) [20:50:43] (03PS1) 10BryanDavis: openstack: Interact with keystone via 'admin' interface [labs/striker] - 10https://gerrit.wikimedia.org/r/352689 (https://phabricator.wikimedia.org/T164787) [20:59:21] 06Labs, 10Striker, 10Tool-Labs, 13Patch-For-Review, 15User-bd808: Keystone permissions exception when trying to approve Tool Labs membership request via Striker - https://phabricator.wikimedia.org/T164787#3245640 (10bd808) I tested the patch from https://gerrit.wikimedia.org/r/352689 by cherry-picking on... [20:59:43] RECOVERY - Puppet errors on tools-exec-1437 is OK: OK: Less than 1.00% above the threshold [0.0] [21:23:59] 10Labs-project-Wikistats: Add pa.wikisource to WikiStats - https://phabricator.wikimedia.org/T164798#3245756 (10Dereckson) [21:24:18] 10Labs-project-Wikistats: Add pa.wikisource to WikiStats - https://phabricator.wikimedia.org/T164798#3245771 (10Dereckson) [21:25:41] 06Labs, 10Striker, 10Tool-Labs, 13Patch-For-Review, 15User-bd808: Keystone permissions exception when trying to approve Tool Labs membership request via Striker - https://phabricator.wikimedia.org/T164787#3245790 (10bd808) It turns out that the poorly documented trick needed is that when authenticating t... [21:46:33] (03PS2) 10BryanDavis: openstack: Role modifications require global admin rights [labs/striker] - 10https://gerrit.wikimedia.org/r/352689 (https://phabricator.wikimedia.org/T164787) [21:51:35] (03CR) 10BryanDavis: [V: 032] "Tested via cherry-pick on californium" [labs/striker] - 10https://gerrit.wikimedia.org/r/352689 (https://phabricator.wikimedia.org/T164787) (owner: 10BryanDavis) [21:52:51] Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/Cookies52 was modified, changed by Cookies52 link https://wikitech.wikimedia.org/w/index.php?diff=1758692 edit summary: cmt [21:56:05] (03CR) 10Andrew Bogott: [C: 04-1] openstack: Role modifications require global admin rights (031 comment) [labs/striker] - 10https://gerrit.wikimedia.org/r/352689 (https://phabricator.wikimedia.org/T164787) (owner: 10BryanDavis) [21:58:48] (03CR) 10Andrew Bogott: [C: 04-1] openstack: Role modifications require global admin rights (031 comment) [labs/striker] - 10https://gerrit.wikimedia.org/r/352689 (https://phabricator.wikimedia.org/T164787) (owner: 10BryanDavis) [22:13:20] 06Labs, 15User-bd808: Remove 2FA on wikitech for user Mdann52 - https://phabricator.wikimedia.org/T164804#3245920 (10bd808) [22:15:14] 06Labs, 15User-bd808: Remove 2FA on wikitech for user Mdann52 - https://phabricator.wikimedia.org/T164804#3245934 (10bd808) Ownership of account verified via file in homedir on tool labs: ``` tools-bastion-02.tools:~ bd808$ sudo ls -al ~mdann52/2fa -rw-r--r-- 1 mdann52 wikidev 49 May 8 21:51 /home/mdann52/2fa... [22:18:10] 06Labs, 15User-bd808: Remove 2FA on wikitech for user Mdann52 - https://phabricator.wikimedia.org/T164804#3245937 (10bd808) 05Open>03Resolved [22:20:15] Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/Cookies52 was modified, changed by BryanDavis link https://wikitech.wikimedia.org/w/index.php?diff=1758696 edit summary: [[phab:T164804]] [22:24:57] (03PS3) 10BryanDavis: openstack: Role modifications require global admin rights [labs/striker] - 10https://gerrit.wikimedia.org/r/352689 (https://phabricator.wikimedia.org/T164787) [22:26:42] (03CR) 10Andrew Bogott: [C: 032] openstack: Role modifications require global admin rights [labs/striker] - 10https://gerrit.wikimedia.org/r/352689 (https://phabricator.wikimedia.org/T164787) (owner: 10BryanDavis) [22:28:29] (03Merged) 10jenkins-bot: openstack: Role modifications require global admin rights [labs/striker] - 10https://gerrit.wikimedia.org/r/352689 (https://phabricator.wikimedia.org/T164787) (owner: 10BryanDavis) [22:32:49] 10Labs-project-Wikistats: Add pa.wikisource to WikiStats - https://phabricator.wikimedia.org/T164798#3245956 (10Dzahn) a:03Dzahn [22:46:02] (03PS1) 10BryanDavis: Bump striker submodule [labs/striker/deploy] - 10https://gerrit.wikimedia.org/r/352719 (https://phabricator.wikimedia.org/T164787) [22:46:25] (03CR) 10BryanDavis: [C: 032] Bump striker submodule [labs/striker/deploy] - 10https://gerrit.wikimedia.org/r/352719 (https://phabricator.wikimedia.org/T164787) (owner: 10BryanDavis) [22:46:35] (03Merged) 10jenkins-bot: Bump striker submodule [labs/striker/deploy] - 10https://gerrit.wikimedia.org/r/352719 (https://phabricator.wikimedia.org/T164787) (owner: 10BryanDavis) [22:47:03] 06Labs, 10Labs-Infrastructure: labservices1002 mostly won't boot - https://phabricator.wikimedia.org/T164807#3245993 (10Andrew) [23:04:48] 06Labs, 10Striker, 10Tool-Labs, 13Patch-For-Review, 15User-bd808: Implement Tool Labs membership application and processing in Striker - https://phabricator.wikimedia.org/T162508#3246054 (10bd808) [23:04:50] 06Labs, 10Striker, 10Tool-Labs, 13Patch-For-Review, 15User-bd808: Keystone permissions exception when trying to approve Tool Labs membership request via Striker - https://phabricator.wikimedia.org/T164787#3246053 (10bd808) 05Open>03Resolved [23:13:05] 06Labs, 10wikitech.wikimedia.org: Old versions of Semantic* extensions break Wikitech (after removing long-living deprecated functions from core) - https://phabricator.wikimedia.org/T123599#3246066 (10bd808) [23:13:10] 06Labs, 10Labs-Infrastructure, 10Tool-Labs, 10MediaWiki-extensions-Page_Forms, 13Patch-For-Review: https://wikitech.wikimedia.org/wiki/Special:FormEdit/Tools_Access_Request down - https://phabricator.wikimedia.org/T123583#3246068 (10bd808) [23:13:15] 06Labs, 10Tool-Labs, 06Project-Admins: Migrate Tools access request process to Phabricator - https://phabricator.wikimedia.org/T72625#3246064 (10bd808) 05Open>03declined {T162508} is deployed now. Lets try that out for a while. If it turns out to be awkward we can revisit this workflow idea as a replacem... [23:15:25] 06Labs, 10Labs-Infrastructure: labservices1002 slow puppet runs and IO issues - https://phabricator.wikimedia.org/T164675#3246074 (10Andrew) [23:15:27] 06Labs, 10Labs-Infrastructure: labservices1002 mostly won't boot - https://phabricator.wikimedia.org/T164807#3246072 (10Andrew) 05Open>03Resolved This is fixed thanks to dzahn's help and a variation on http://crtech.tips/12g-lifecycle-controller-hangs-during-post/ I rebooted from the console and it came b... [23:15:51] 06Labs, 10Labs-Infrastructure: labservices1002 slow puppet runs and IO issues - https://phabricator.wikimedia.org/T164675#3241620 (10Andrew) I've changed the bios setting in question, so /maybe/ this is resolved. I'm going to keep an eye out for a few days though. [23:18:30] (03CR) 10Andrew Bogott: [V: 032 C: 032] Added dummy striker creds [labs/private] - 10https://gerrit.wikimedia.org/r/352644 (owner: 10Andrew Bogott) [23:28:44] 06Labs, 10wikitech.wikimedia.org, 05Goal, 13Patch-For-Review: Get rid of SemanticMediaWiki/SRF/SF from wikitech.wikimedia.org - https://phabricator.wikimedia.org/T53642#3246085 (10bd808) * Deleted Form:Nova_Project_Documentation * Deleted Form:Server * Deleted Form:Tools_Access_Request/Edit * Deleted Form:... [23:31:12] 06Labs, 10Labs-Infrastructure: labservices1002 slow puppet runs and IO issues - https://phabricator.wikimedia.org/T164675#3246092 (10faidon) >>! In T164675#3246074, @Andrew wrote: > I've changed the bios setting in question, so /maybe/ this is resolved. I'm going to keep an eye out for a few days though. Gre...