[00:05:22] Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/Drewmutt was created, changed by Drewmutt link https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Access_Request/Drewmutt edit summary: Created page with "{{Tools Access Request |Justification=I'm an experienced developer and would like to help WM develop some helpful tools. |Completed=false |User Name=Drewmutt }}" [01:03:06] PROBLEM - Puppet run on tools-exec-1401 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [01:31:18] 06Labs, 10Tool-Labs, 10Tools-Kubernetes, 07Tracking: Packages to be installed in Tool Labs Kubernetes Images (Tracking) - https://phabricator.wikimedia.org/T140110#3127344 (10zhuyifei1999) [01:31:20] 06Labs, 10Tool-Labs: Add dependencies for Postgresql to Kubernetes container - https://phabricator.wikimedia.org/T161266#3127343 (10zhuyifei1999) [01:43:07] RECOVERY - Puppet run on tools-exec-1401 is OK: OK: Less than 1.00% above the threshold [0.0] [06:37:32] PROBLEM - Puppet run on tools-exec-1414 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [06:47:29] PROBLEM - Puppet run on tools-exec-1409 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [06:58:11] PROBLEM - Puppet run on tools-webgrid-generic-1401 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [06:58:25] 06Labs, 10Tool-Labs, 06Community-Tech-Tool-Labs, 10Tools-Kubernetes: My first kubernetes + python3 + django app tutorial - https://phabricator.wikimedia.org/T149191#3127479 (10Tobias1984) @bd808 I would still like to leave this task open until the workshop at the Prague and Vienna hackathon is over to see... [07:17:33] RECOVERY - Puppet run on tools-exec-1414 is OK: OK: Less than 1.00% above the threshold [0.0] [07:22:28] RECOVERY - Puppet run on tools-exec-1409 is OK: OK: Less than 1.00% above the threshold [0.0] [07:33:09] RECOVERY - Puppet run on tools-webgrid-generic-1401 is OK: OK: Less than 1.00% above the threshold [0.0] [14:26:49] 06Labs, 10The-Wikipedia-Library: Requesting /data/project NFS share for Nova_Resource:Twl - https://phabricator.wikimedia.org/T159407#3128216 (10jsn.sherman) @bd808 @chasemp is there any other information you need? Or do we need to adjust our expectations/plan regarding backup? We landed on requesting this bec... [14:57:31] 06Labs, 10Tool-Labs, 06Community-Tech-Tool-Labs, 10Tools-Kubernetes: My first kubernetes + python3 + django app tutorial - https://phabricator.wikimedia.org/T149191#3128278 (10bd808) >>! In T149191#3127479, @Tobias1984 wrote: > @bd808 I would still like to leave this task open until the workshop at the Pra... [15:19:04] 06Labs, 10Tool-Labs, 06Community-Tech-Tool-Labs, 10Tools-Kubernetes: My first kubernetes + python3 + django app tutorial - https://phabricator.wikimedia.org/T149191#3128317 (10Tobias1984) > It would be nice to see instructions on using the shared MariaDB/MySQL database servers as the data store instead. E... [15:28:23] 06Labs, 06DC-Ops, 06Operations: Move labstore1002 and labstore1002-array1 and labstore1002-array2 to different rack (currently in C3) - https://phabricator.wikimedia.org/T158913#3128323 (10madhuvishy) @Cmjohnson Bumping this, we'd like to move forward with this soon if possible :) Thanks! [15:28:58] 06Labs, 10The-Wikipedia-Library: Requesting /data/project NFS share for Nova_Resource:Twl - https://phabricator.wikimedia.org/T159407#3128324 (10bd808) p:05Triage>03Normal a:03madhuvishy Assigning to @madhuvishy for the backend puppet changes needed. @jsn.sherman or @Samwalton9, can you update the task... [15:41:23] 06Labs, 10Datasets-General-or-Unknown, 10Dumps-Generation, 06Operations, 10hardware-requests: Eqiad: Hardware request for labstore1006/7, dataset1002/3 - https://phabricator.wikimedia.org/T161311#3128341 (10ArielGlenn) [15:44:17] 06Labs, 10The-Wikipedia-Library: Requesting /data/project NFS share for Nova_Resource:Twl - https://phabricator.wikimedia.org/T159407#3128378 (10jsn.sherman) Right now we just need: Twlight-test.twl.eqiad.wmflabs I anticipate having 3 hosts in total in the future, but I will try to batch those changes so it c... [16:00:05] 06Labs, 06Operations: Instance creation fails before first puppet run around 1% of the time - https://phabricator.wikimedia.org/T160908#3128433 (10chasemp) Took about a day and half to leak 7 instances https://graphite.wikimedia.org/render/?width=586&height=308&_salt=1486132582.09&target=cactiStyle(log(server... [16:09:20] 06Labs, 06DC-Ops, 06Operations: Move labstore1002 and labstore1002-array1 and labstore1002-array2 to different rack (currently in C3) - https://phabricator.wikimedia.org/T158913#3128457 (10Cmjohnson) @madhuvishy . Sorry I was away on vacation. When do you want to do this next week. I would prefer if we coul... [16:12:12] 06Labs, 06DC-Ops, 06Operations: Move labstore1002 and labstore1002-array1 and labstore1002-array2 to different rack (currently in C3) - https://phabricator.wikimedia.org/T158913#3128465 (10madhuvishy) @Cmjohnson Yup no problem! :) Just to confirm, can we move both servers to row B then? [16:23:58] 06Labs, 06DC-Ops, 06Operations: Move labstore1002 and labstore1002-array1 and labstore1002-array2 to different rack (currently in C3) - https://phabricator.wikimedia.org/T158913#3128477 (10Cmjohnson) @madhuvishy You want both moved to row B but separate racks? If they need to be connected to each other that... [16:29:15] 06Labs, 06DC-Ops, 06Operations: Move labstore1002 and labstore1002-array1 and labstore1002-array2 to different rack (currently in C3) - https://phabricator.wikimedia.org/T158913#3128493 (10madhuvishy) @Cmjohnson Yes I think separate racks for reliability. We currently have labstore1004 and labstore1005 set u... [16:30:40] 06Labs, 06DC-Ops, 06Operations: Move labstore1002 and labstore1002-array1 and labstore1002-array2 to different rack (currently in C3) - https://phabricator.wikimedia.org/T158913#3128506 (10chasemp) It's similar to what we are doing with labstore1004/1005, or same model. I believe those are in neighboring ra... [16:44:03] 06Labs, 10Tool-Labs: Cannot access replica databases - access denied - https://phabricator.wikimedia.org/T151296#2813451 (10madhuvishy) @MnemonicFlow Can you check if this works now? [17:00:42] 06Labs, 10Tool-Labs: Cannot access replica databases - access denied - https://phabricator.wikimedia.org/T151296#3128570 (10MnemonicFlow) No. I cannot connect. I've tried with my shell user: mnemonicflow, doesn't work. -rw-r--r-- 1 mnemonicflow wikidev 50 nov 18 08:17 my.cnf -rw-r--r-- 1 mnemonicflow wik... [17:15:06] 06Labs, 10Tool-Labs: Cannot access replica databases - access denied - https://phabricator.wikimedia.org/T151296#2813451 (10jcrespo) @madhuvishy I see that the user exists: ``` root@labsdb1001[(none)]> SHOW GRANTS FOR 'u4507'; +---------------------------------------------------------------+ | Grants for u45... [17:29:59] 06Labs, 10Labs-Infrastructure, 10DNS, 06Discovery, and 3 others: multi-component wmflabs.org subdomains doesn't work under simple wildcard TLS cert - https://phabricator.wikimedia.org/T161256#3128645 (10MaxSem) These subdomains should just be removed. They made sense in the times of HTTP when browsers had... [17:30:45] 06Labs, 10Tool-Labs: Cannot access replica databases - access denied - https://phabricator.wikimedia.org/T151296#3128648 (10madhuvishy) Okay I think this springs from there being two users cff and mnemonicflow and both mapping to ldap user id 4507 - I'm looking into how we can resolve this. [17:44:18] 06Labs, 10Tool-Labs: Cannot access replica databases - access denied - https://phabricator.wikimedia.org/T151296#3128728 (10madhuvishy) @MnemonicFlow Hi, okay - i'm still investigating how to cleanup user cff. But I think your replica file as shell user MnemonicFlow on tools should work now. [17:49:32] marxarelli: I'm available for about an hour now again if you want any k8s help :) [17:55:20] 06Labs, 10Tool-Labs: Cannot access replica databases - access denied - https://phabricator.wikimedia.org/T151296#3128755 (10madhuvishy) Update - both shell accounts cff and MnemonicFlow should have working replica.my.cnfs in their home directory now. [18:10:28] 06Labs, 10Tool-Labs, 10Prod-Kubernetes, 10Tools-Kubernetes, 07kubernetes: Fully document process for building a new version of Kubernetes debs - https://phabricator.wikimedia.org/T161031#3128798 (10yuvipanda) @akosiaris progress! but now stuck at: ``` dpkg-source: info: the patch has fuzz which is not a... [18:46:33] 06Labs, 10Labs-Infrastructure: bootstrap_vz: Move firstboot.sh out of the base image? - https://phabricator.wikimedia.org/T161327#3128899 (10Andrew) [18:47:32] 06Labs, 10Labs-Infrastructure: bootstrap_vz: Move firstboot.sh out of the base image? - https://phabricator.wikimedia.org/T161327#3128913 (10Andrew) (And we should have some kind of validation for the script, probably by putting a hash in the nova metadata.) [19:29:05] 10Labs-project-Wikistats: Language name in rank.php should show up the language name in that language, not in English. - https://phabricator.wikimedia.org/T111607#3128963 (10Dzahn) Should it show _only_ the name of the language in itself or should it show both, local name and English name? [20:18:40] 10Labs-project-Wikistats, 13Patch-For-Review: Language name in rank.php should show up the language name in that language, not in English. - https://phabricator.wikimedia.org/T111607#3129139 (10Dzahn) 05Open>03Resolved @revi This is now done. Language names in drop-down menu are using local names. [20:18:43] 10Labs-project-Wikistats: Language name in rank.php should show up the language name in that language, not in English. - https://phabricator.wikimedia.org/T111607#3129141 (10Dzahn) [21:04:00] 06Labs, 10Horizon, 07Developer-notice: Horizon Mitaka 'remember me' checkbox immune to keyboard focus - https://phabricator.wikimedia.org/T158103#3129212 (10Andrew) p:05Triage>03Low [21:04:21] 06Labs, 10Labs-Infrastructure, 13Patch-For-Review: Rebalance tools exec nodes with an eye towards CPU usage - https://phabricator.wikimedia.org/T161006#3129213 (10Andrew) p:05Triage>03Normal [21:04:38] 06Labs, 10Labs-Infrastructure: bootstrap_vz: Move firstboot.sh out of the base image? - https://phabricator.wikimedia.org/T161327#3129215 (10Andrew) p:05Triage>03Normal [21:05:41] 06Labs, 13Patch-For-Review: check on the nova-api upstart logs - https://phabricator.wikimedia.org/T159141#3129222 (10Andrew) 05Open>03Resolved This seems to actually work properly now. [21:08:46] 06Labs, 10Labs-Infrastructure, 06Operations, 10ops-eqiad: Labvirt1001 has insanely slow IO - https://phabricator.wikimedia.org/T159835#3129247 (10Andrew) The current state of this is: I rebooted labvirt1001 and it got better. I've migrated a handful of tools exec nodes back to labvirt1001 and I'm going t... [21:10:42] 06Labs, 10Labs-Infrastructure, 13Patch-For-Review: Convince nova-scheduler to pay attention to CPU metrics - https://phabricator.wikimedia.org/T161006#3129254 (10Andrew) [21:32:24] 06Labs, 10Datasets-General-or-Unknown, 10Dumps-Generation, 06Operations, 10hardware-requests: Eqiad: Hardware request for labstore1006/7, dataset1002/3 - https://phabricator.wikimedia.org/T161311#3129356 (10ArielGlenn)