[01:17:18] (03PS2) 10BryanDavis: [WIP] jsub: Remove support for release=precise [labs/toollabs] - 10https://gerrit.wikimedia.org/r/341666 (https://phabricator.wikimedia.org/T94792) [01:18:37] (03CR) 10jerkins-bot: [V: 04-1] [WIP] jsub: Remove support for release=precise [labs/toollabs] - 10https://gerrit.wikimedia.org/r/341666 (https://phabricator.wikimedia.org/T94792) (owner: 10BryanDavis) [02:37:46] (03CR) 10Zhuyifei1999: Add rewritten crontab in Python (033 comments) [labs/toollabs] - 10https://gerrit.wikimedia.org/r/336998 (https://phabricator.wikimedia.org/T156174) (owner: 10Zhuyifei1999) [04:12:49] (03CR) 10BryanDavis: [C: 04-1] Add rewritten crontab in Python (035 comments) [labs/toollabs] - 10https://gerrit.wikimedia.org/r/336998 (https://phabricator.wikimedia.org/T156174) (owner: 10Zhuyifei1999) [06:29:14] (03PS3) 10BryanDavis: jsub: Remove support for release=precise [labs/toollabs] - 10https://gerrit.wikimedia.org/r/341666 (https://phabricator.wikimedia.org/T94792) [06:31:41] (03CR) 10BryanDavis: jsub: Remove support for release=precise (031 comment) [labs/toollabs] - 10https://gerrit.wikimedia.org/r/341666 (https://phabricator.wikimedia.org/T94792) (owner: 10BryanDavis) [06:48:21] PROBLEM - Puppet run on tools-exec-1410 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [07:06:00] PROBLEM - Puppet run on tools-exec-1403 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [07:12:38] PROBLEM - Puppet run on tools-exec-1406 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [07:23:22] RECOVERY - Puppet run on tools-exec-1410 is OK: OK: Less than 1.00% above the threshold [0.0] [07:45:59] RECOVERY - Puppet run on tools-exec-1403 is OK: OK: Less than 1.00% above the threshold [0.0] [07:52:38] RECOVERY - Puppet run on tools-exec-1406 is OK: OK: Less than 1.00% above the threshold [0.0] [08:03:36] Hi, is there a known issue with account creation on labs. https://phabricator.wikimedia.org/T149109 [08:04:02] A student of mine is getting the following error [08:04:35] thus I created the account for him, but he could not change his temorary generated password [08:04:46] Password change denied [08:54:54] PROBLEM - Puppet run on tools-webgrid-generic-1402 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [09:29:55] RECOVERY - Puppet run on tools-webgrid-generic-1402 is OK: OK: Less than 1.00% above the threshold [0.0] [10:17:55] 06Labs, 10Labs-Infrastructure, 10DBA, 06Operations: labsdb1006/1007 (postgresql) maintenance - https://phabricator.wikimedia.org/T157359#3083531 (10Marostegui) Shall I start copying labsdb1007 to dbstore1001 or are you "breaking" at the moment it @jcrespo? [10:18:37] 06Labs, 10Labs-Infrastructure, 10DBA, 06Operations: labsdb1006/1007 (postgresql) maintenance - https://phabricator.wikimedia.org/T157359#3083532 (10jcrespo) We can start this now. I was about to do it. [10:22:29] 06Labs, 10Labs-Infrastructure, 10DBA, 06Operations: labsdb1006/1007 (postgresql) maintenance - https://phabricator.wikimedia.org/T157359#3083581 (10Marostegui) >>! In T157359#3083532, @jcrespo wrote: > We can start this now. I was about to do it. Ah, go ahead if you like then. [10:23:54] 06Labs, 10DBA, 13Patch-For-Review: labsdb1004 MySQL crash - https://phabricator.wikimedia.org/T159572#3083591 (10jcrespo) I am going to drop s51412__data from labsdb1004 only , and the others filtered, to avoid confusion on where data is up to date. [10:29:21] 06Labs, 10Labs-Infrastructure, 10DBA, 06Operations: labsdb1006/1007 (postgresql) maintenance - https://phabricator.wikimedia.org/T157359#3083645 (10jcrespo) I am with labsdb1004 now, please shutdown postgres and copy it. [10:30:02] 06Labs, 10Labs-Infrastructure, 10DBA, 06Operations: labsdb1006/1007 (postgresql) maintenance - https://phabricator.wikimedia.org/T157359#3083647 (10Marostegui) Oki doki! [10:37:46] 06Labs, 10Labs-Infrastructure, 10DBA, 06Operations: labsdb1006/1007 (postgresql) maintenance - https://phabricator.wikimedia.org/T157359#3083686 (10Marostegui) Transfer started and the file will be located at: `dbstore1001:/srv/tmp/labsdb1007.tar.gz` [10:42:36] 06Labs, 10DBA, 13Patch-For-Review: labsdb1004 MySQL crash - https://phabricator.wikimedia.org/T159572#3083690 (10jcrespo) 05Open>03Resolved This is fixed for labsdb1004, except for https://gerrit.wikimedia.org/r/341551 , and labsdb1005, which requires a restart to apply new changes. [10:48:45] 06Labs, 15User-Hydriz: Create custom instance flavor for Dumps project - https://phabricator.wikimedia.org/T159930#3083694 (10Hydriz) [11:16:32] 06Labs, 10Labs-Infrastructure, 10DBA, 06Operations, 13Patch-For-Review: labsdb1006/1007 (postgresql) maintenance - https://phabricator.wikimedia.org/T157359#3083756 (10Marostegui) `labsdb1007:/etc` directory has been copied to `dbstore1001:/srv/tmp/labsdb1007_etc.tar.gz` [12:03:29] 06Labs, 10Tool-Labs, 06Community-Tech-Tool-Labs, 07Documentation: Create a "my first Pywikibot bot" tutorial for Tool Labs - https://phabricator.wikimedia.org/T134495#3083840 (10Xephyr826) Thanks! I'll come back to this next week when I have more time. [12:03:49] 06Labs, 10Tool-Labs, 06Community-Tech-Tool-Labs, 07Documentation: Create a "my first Pywikibot bot" tutorial for Tool Labs - https://phabricator.wikimedia.org/T134495#3083841 (10Xephyr826) a:05Xephyr826>03None [12:08:08] 06Labs, 10Labs-Infrastructure, 10DBA, 06Operations, 13Patch-For-Review: labsdb1006/1007 (postgresql) maintenance - https://phabricator.wikimedia.org/T157359#3002535 (10ops-monitoring-bot) Script wmf_auto_reimage was launched by jynus on neodymium.eqiad.wmnet for hosts: ``` ['labsdb1007.eqiad.wmnet'] ```... [12:15:19] 06Labs, 10Labs-Infrastructure, 10DBA, 06Operations, 13Patch-For-Review: labsdb1006/1007 (postgresql) maintenance - https://phabricator.wikimedia.org/T157359#3083856 (10jcrespo) The installer ask for verification to delete all partitions- this should be changed on the recipe (or use the db one, where this... [12:34:17] 06Labs, 10Labs-Infrastructure, 10DBA, 06Operations, 13Patch-For-Review: labsdb1006/1007 (postgresql) maintenance - https://phabricator.wikimedia.org/T157359#3083907 (10ops-monitoring-bot) Completed auto-reimage of hosts: ``` ['labsdb1007.eqiad.wmnet'] ``` and were **ALL** successful. [12:40:33] 06Labs, 10Labs-Infrastructure, 10DBA, 06Operations, 13Patch-For-Review: labsdb1006/1007 (postgresql) maintenance - https://phabricator.wikimedia.org/T157359#3083935 (10jcrespo) It installed ok, but /dev/tank/data has only 500GB. How was that partitioned @akosiaris ? [12:54:13] 06Labs, 10Labs-Infrastructure, 10DBA, 06Operations, 13Patch-For-Review: labsdb1006/1007 (postgresql) maintenance - https://phabricator.wikimedia.org/T157359#3083946 (10akosiaris) It's missing the rest of the disks in the md RAID5 (and RAID1 for /boot) for some reason. Maybe some difference between jessie... [13:02:37] 06Labs, 10Labs-Infrastructure, 10DBA, 06Operations, 13Patch-For-Review: labsdb1006/1007 (postgresql) maintenance - https://phabricator.wikimedia.org/T157359#3083951 (10akosiaris) Nope. not that. What's actually the culprit is rOPUP1b3633e where labsdb1007 gets matched incorrectly [13:11:07] 06Labs, 10Labs-Infrastructure, 10DBA, 06Operations, 13Patch-For-Review: labsdb1006/1007 (postgresql) maintenance - https://phabricator.wikimedia.org/T157359#3083958 (10jcrespo) Thanks, I can amend that and try again. [13:31:46] 06Labs, 10Labs-Infrastructure, 10DBA, 06Operations, 13Patch-For-Review: labsdb1006/1007 (postgresql) maintenance - https://phabricator.wikimedia.org/T157359#3083998 (10Marostegui) Merged both patches from @jcrespo and ran puppet on install1002, going to try to reimage the server again [13:33:05] 10Tool-Labs-tools-Other, 07Tracking: merl tools (tracking) - https://phabricator.wikimedia.org/T69556#732994 (10Liuxinyu970226) >>! In T69556#733019, @Merl wrote: > It is a tracking bug for features missing at labs, mediawiki, wikidata or any other component covered by this bugzilla that any of my tools relies... [13:33:24] 06Labs, 10Labs-Infrastructure, 10DBA, 06Operations, 13Patch-For-Review: labsdb1006/1007 (postgresql) maintenance - https://phabricator.wikimedia.org/T157359#3084004 (10ops-monitoring-bot) Script wmf_auto_reimage was launched by marostegui on neodymium.eqiad.wmnet for hosts: ``` ['labsdb1007.eqiad.wmnet']... [14:43:48] 06Labs, 10Labs-Infrastructure, 10DBA, 06Operations, 13Patch-For-Review: labsdb1006/1007 (postgresql) maintenance - https://phabricator.wikimedia.org/T157359#3084272 (10Marostegui) Looks good now: ``` root@labsdb1007:~# pvs PV VG Fmt Attr PSize PFree /dev/md1 labsdb1007-vg lvm2 a... [14:46:43] 06Labs, 10wikitech.wikimedia.org, 15User-bd808: Account creation failing with "The authentication plugin denied the account creation." - https://phabricator.wikimedia.org/T149109#3084275 (10Physikerwelt) 05Resolved>03Open A student of mine is getting the following error "The authentication plugin denied... [14:48:10] 06Labs, 10Labs-Infrastructure, 10DBA, 06Operations, 13Patch-For-Review: labsdb1006/1007 (postgresql) maintenance - https://phabricator.wikimedia.org/T157359#3084282 (10ops-monitoring-bot) Completed auto-reimage of hosts: ``` ['labsdb1007.eqiad.wmnet'] ``` and were **ALL** successful. [15:12:28] PROBLEM - Puppet run on tools-checker-02 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [15:22:46] (03CR) 10Zhuyifei1999: "I'll do the changes on Friday" (032 comments) [labs/toollabs] - 10https://gerrit.wikimedia.org/r/336998 (https://phabricator.wikimedia.org/T156174) (owner: 10Zhuyifei1999) [15:32:53] 06Labs, 10Labs-Infrastructure, 10DBA, 06Operations, 13Patch-For-Review: labsdb1006/1007 (postgresql) maintenance - https://phabricator.wikimedia.org/T157359#3084373 (10Marostegui) So, for the record we saw something with the logical volume: ``` root@labsdb1007:~# lvs LV VG Attr... [15:47:27] RECOVERY - Puppet run on tools-checker-02 is OK: OK: Less than 1.00% above the threshold [0.0] [16:25:16] 06Labs, 10wikitech.wikimedia.org: Account creation failing with "The authentication plugin denied the account creation." - https://phabricator.wikimedia.org/T149109#3084534 (10bd808) a:05bd808>03None I can recreate the failure (attempted username: Bd20170308; shell name: bd170308). * UI message given is "... [16:39:42] 06Labs, 10wikitech.wikimedia.org: Account creation failing with "The authentication plugin denied the account creation." - https://phabricator.wikimedia.org/T149109#3084567 (10bd808) I can't find the `VincSohn` and `Qwery0` accounts that have been created by logged in users in LDAP. [16:50:55] PROBLEM - Puppet run on tools-webgrid-generic-1402 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [16:57:37] (03CR) 10BryanDavis: [C: 04-1] Add rewritten crontab in Python (031 comment) [labs/toollabs] - 10https://gerrit.wikimedia.org/r/336998 (https://phabricator.wikimedia.org/T156174) (owner: 10Zhuyifei1999) [17:01:51] 06Labs, 10wikitech.wikimedia.org: Account creation failing with "The authentication plugin denied the account creation." - https://phabricator.wikimedia.org/T149109#3084604 (10bd808) On irc @tgr noted that this problem existed on 2017-03-04 as well when he created an account for a user at a hackathon. The last... [17:30:54] RECOVERY - Puppet run on tools-webgrid-generic-1402 is OK: OK: Less than 1.00% above the threshold [0.0] [17:51:56] 06Labs, 10DBA, 13Patch-For-Review: Add and sanitize s2, s4, s5, s6 and s7 to sanitarium2 and new labsdb hosts - https://phabricator.wikimedia.org/T153743#3084728 (10Marostegui) db1070 has been restarted with ROW based replication to serve as a master for db1095. [19:00:10] 06Labs, 10wikitech.wikimedia.org: Account creation failing with "The authentication plugin denied the account creation." - https://phabricator.wikimedia.org/T149109#3084994 (10bd808) p:05Triage>03High [19:02:50] 06Labs, 06Operations, 07Tracking: Cleanup tools nfs share on labstore1004/5 - https://phabricator.wikimedia.org/T156982#3085020 (10madhuvishy) Doing this again today: Truncating log/err/out files over 100M. List of files: ``` 101G /srv/tools/shared/tools/project/whymbot/enwikt.err 93G /srv/tools/shared/tool... [19:29:59] 06Labs, 10Tool-Labs, 06Community-Tech-Tool-Labs, 10Tools-Kubernetes: My first kubernetes + python3 + django app tutorial - https://phabricator.wikimedia.org/T149191#3085154 (10Tobias1984) [19:38:47] 06Labs, 10wikitech.wikimedia.org, 13Patch-For-Review: Account creation failing with "The authentication plugin denied the account creation." - https://phabricator.wikimedia.org/T149109#3085223 (10bd808) Verbose LDAP log from a failed attempt ``` 2017-03-08 19:33:09 [5a12e625aa950ea368857c4d] silver labswiki... [19:44:53] 06Labs, 10wikitech.wikimedia.org, 13Patch-For-Review: Account creation failing with "The authentication plugin denied the account creation." - https://phabricator.wikimedia.org/T149109#3085271 (10bd808) 05Open>03Resolved a:03Andrew "Welcome, Bd20170308!" -- account creation fixed after @andrew raised t... [19:45:41] Why can't I ever get the fcgi example to work? [19:46:18] Its like the code was never tested https://wikitech.wikimedia.org/wiki/Help:Tool_Labs/Web#Example_configurations [19:51:41] 10Striker: Check to see if Striker can properly create accounts when ldap results are limited - https://phabricator.wikimedia.org/T159981#3085302 (10bd808) [20:00:45] 10Tool-Labs-tools-Pageviews: Add ability to query for legacy pageviews - https://phabricator.wikimedia.org/T149358#3085381 (10MusikAnimal) [20:00:53] 10Tool-Labs-tools-Pageviews: Add ability to query for legacy pageviews - https://phabricator.wikimedia.org/T149358#2749796 (10MusikAnimal) [20:17:44] 06Labs: Making access to index of table backlinks - https://phabricator.wikimedia.org/T159984#3085420 (10Ebraminio) [20:18:44] 06Labs: Providing index of backlinks table to labs replicas - https://phabricator.wikimedia.org/T159984#3085434 (10Ebraminio) [20:21:19] 06Labs: Providing index of backlinks table to labs replicas - https://phabricator.wikimedia.org/T159984#3085442 (10Ebraminio) [20:30:48] 06Labs, 10Tool-Labs, 10Tools-Kubernetes: Reassign service/pod IP ranges for kubernetes on tool labs - https://phabricator.wikimedia.org/T152399#3085461 (10chasemp) Outline of end to end test of this version 1: root@k8s-master-01:~# kubectl get deployments --all-namespaces NAMESPACE NAME DESIRED C... [20:36:04] 06Labs, 10Tool-Labs, 10Tools-Kubernetes: Reassign service/pod IP ranges for kubernetes on tool labs - https://phabricator.wikimedia.org/T152399#3085489 (10chasemp) [20:42:59] 06Labs, 10Labs-Infrastructure, 10DBA, 06Operations, 13Patch-For-Review: labsdb1006/1007 (postgresql) maintenance - https://phabricator.wikimedia.org/T157359#3085505 (10jcrespo) > Hey, I am not saying it is going to work 100% sure- I am just suggesting to try it first, and then go the slow route, which is... [20:53:59] 06Labs, 10Tool-Labs, 10Tools-Kubernetes: Reassign service/pod IP ranges for kubernetes on tool labs - https://phabricator.wikimedia.org/T152399#3085521 (10yuvipanda) This looks good, the thing we need to test is if the service IPs are actually reassigned automatically if we change the service IP range. We cu... [20:56:52] 06Labs, 10wikitech.wikimedia.org, 15User-bd808: Fix or delete user accounts showing on wikitech but not found in LDAP after T149109 - https://phabricator.wikimedia.org/T159986#3085528 (10bd808) [20:57:35] 06Labs, 10wikitech.wikimedia.org, 15User-bd808: Fix or delete user accounts showing on wikitech but not found in LDAP after T149109 - https://phabricator.wikimedia.org/T159986#3085544 (10bd808) p:05Triage>03High [21:07:34] Hi, it seems one of the instances i just rebooted [21:07:46] is not starting [21:07:50] failing with [21:07:51] [ 22.807223] init: Failed to obtain startpar-bridge instance: Unknown parameter: INSTANCE [21:07:58] the instance is gerrit-test3 [21:08:45] Full log [21:08:46] https://phabricator.wikimedia.org/P5031 [21:08:54] chasemp ^^ [21:09:15] andrewbogott: I'm on a call^ any chance you can look [21:11:29] It did a kernal update for 3.16 just before i rebooted. But the instance is running 4.4. [21:18:12] andrewbogott ^^ [21:22:03] cloud-init-nonet[152.95]: gave up waiting for a network device. [21:22:09] paladox: what do you mean by 'for 3.16 but is running 4.4'? [21:22:19] paladox: what project and instance name? [21:22:27] oh, gerrit-test3 [21:22:30] in project 'gerrit'? [21:22:36] project is git [21:22:41] and instance is gerrit-test3 [21:22:47] yuvipanda Well i did apt-get update and it said i had an update for 3.16 [21:22:55] but the instance was actually using 4.4 [21:23:21] i just updated gerrit-mysql without any problems [21:23:26] same updating kernal. [21:24:10] paladox: yeah, I think you tried to go from 4.4 to 3.16 and it might've been hosed (From a general googling of the error message you got) [21:24:18] oh [21:24:46] Yeah, I'd expect that to break grub, but I'm not sure that's what's happening here [21:25:18] It seems something is happening with the nework. [21:25:25] according too [21:25:26] cloud-init-nonet[153.12]: gave up waiting for a network device. [21:27:06] https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1341027 [21:27:18] an old bug, but it is to do with kernal 3.16. [21:27:58] paladox: is this urgent? And/or can you just rebuild the instance? [21:28:09] I could rebuild the instance [21:28:29] ok, that's probably the best way forward then. It's hard to know what the apt-get update did, in retrospect. [21:28:51] Ok i will do that now [21:28:52] thanks [21:28:59] sorry I don't have an easier fix. [21:29:07] ok [21:34:26] 06Labs, 10Labs-Infrastructure, 10DBA, 06Operations, 13Patch-For-Review: labsdb1006/1007 (postgresql) maintenance - https://phabricator.wikimedia.org/T157359#3085630 (10MaxSem) Can we use this opportunity to reimport the data from scratch to get rid of possible accumulated OSM replication errors? [21:35:34] How long does it take to build it? It has been 3mins but when i last did it, it was really quick. [21:35:47] andrewbogott ^^ [21:37:50] it's working i think now [21:37:54] from the logs [21:43:52] andrewbogott: thanks, Ididn't realize it was a bad upgrade when I poked you [21:44:53] andrewbogott ok, i've rebuilt it, doing apt-get update not putting a y there, says this [21:44:54] linux-image-3.16.0-4-amd64 [21:45:36] paladox: what's the output of 'uname -r'? [21:45:49] root@gerrit-test3:/home/paladox# uname -r [21:45:49] 4.4.0-3-amd64 [21:45:52] yuvipanda ^^ [21:46:21] Typically we don't upgrade VM kernels... Is there some reason you need to in this case? [21:46:31] Nope [21:46:42] i just like having everything updated with apt-get update [21:47:08] awww [21:47:19] I don't know why apt-get thinks you want a newer 3.16 package [21:47:28] yep [21:48:11] root@gerrit-test3:/home/paladox# apt-cache search linux-image | grep 3.16 [21:48:11] linux-headers-3.16.0-4-amd64 - Header files for Linux 3.16.0-4-amd64 [21:48:11] linux-image-3.16.0-4-amd64 - Linux 3.16 for 64-bit PCs [21:48:12] linux-image-3.16.0-4-amd64-dbg - Debugging symbols for Linux 3.16.0-4-amd64 [21:48:19] it shows it as being installed [21:48:49] oh wait sorry that command searches the archives. [21:51:03] root@gerrit-test3:/home/paladox# sudo update-grub [21:51:04] Generating grub configuration file ... [21:51:04] Warning: Requested serial terminal but GRUB_SERIAL_COMMAND is unspecified. Default parameters will be used. [21:51:04] Found linux image: /boot/vmlinuz-4.4.0-3-amd64 [21:51:05] Found initrd image: /boot/initrd.img-4.4.0-3-amd64 [21:51:05] Found linux image: /boot/vmlinuz-3.16.0-4-amd64 [21:51:06] Found initrd image: /boot/initrd.img-3.16.0-4-amd64 [21:51:08] yuvipanda ^^ [21:51:19] it has two kernals installed it seems [21:52:30] yuvipanda andrewbogott http://askubuntu.com/questions/637038/stop-updating-old-kernel [21:53:02] probaly want to build that in the image or use puppet to prevent the 3.16 kernal from being installed or updated. [21:53:33] paladox: I suggest filing a bug in phabricator, and not running manual upgrades of everything for now [21:53:46] ok [21:56:15] 06Labs, 06Operations: Remove linux kernel 3.16 from the jessie image on labs - https://phabricator.wikimedia.org/T159990#3085707 (10Paladox) [21:56:21] 06Labs, 06Operations: Remove linux kernel 3.16 from the jessie image on labs - https://phabricator.wikimedia.org/T159990#3085719 (10Paladox) p:05Triage>03High [21:56:25] yuvipanda ^^ done [21:58:05] 06Labs, 06Operations: Remove linux kernel 3.16 from the jessie image on labs - https://phabricator.wikimedia.org/T159990#3085707 (10MoritzMuehlenhoff) Why does in break the system if you upgrade the unused 3.16 kernel? [21:58:16] 06Labs, 06Operations: Remove linux kernel 3.16 from the jessie image on labs - https://phabricator.wikimedia.org/T159990#3085725 (10yuvipanda) p:05High>03Normal @MoritzMuehlenhoff any idea what's the right thing to do here? [21:58:36] paladox: I've reset priority, and moritzm should hopefully do things [21:58:43] Ok thanks [21:59:54] 06Labs, 06Operations: Remove linux kernel 3.16 from the jessie image on labs - https://phabricator.wikimedia.org/T159990#3085727 (10Paladox) >>! In T159990#3085723, @MoritzMuehlenhoff wrote: > Why does in break the system if you upgrade the unused 3.16 kernel? It seemed to have broke gerrit-test3. Since after... [22:41:12] (03Draft1) 10Paladox: Gerrit: Add a new $gerrit_auth_email variable [labs/private] - 10https://gerrit.wikimedia.org/r/341918 [22:41:15] (03PS2) 10Paladox: Gerrit: Add a new $gerrit_auth_email variable [labs/private] - 10https://gerrit.wikimedia.org/r/341918 [22:42:11] (03CR) 10Chad: [C: 04-1] "That's what gerrit_email_key already does...." [labs/private] - 10https://gerrit.wikimedia.org/r/341918 (owner: 10Paladox) [22:51:35] !log tools.precise-tools Patched to read from two accounting files [22:51:39] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.precise-tools/SAL [22:58:55] (03PS3) 10Paladox: Gerrit: Have a bogus valus for $gerrit_phab_cert [labs/private] - 10https://gerrit.wikimedia.org/r/341918 [23:00:25] (03CR) 10Chad: "Spelling nit inline, otherwise lgtm" (031 comment) [labs/private] - 10https://gerrit.wikimedia.org/r/341918 (owner: 10Paladox) [23:00:51] (03PS4) 10Paladox: Gerrit: Have a bogus valus for $gerrit_phab_cert [labs/private] - 10https://gerrit.wikimedia.org/r/341918 [23:01:04] (03CR) 10Paladox: Gerrit: Have a bogus valus for $gerrit_phab_cert (031 comment) [labs/private] - 10https://gerrit.wikimedia.org/r/341918 (owner: 10Paladox) [23:05:35] (03CR) 10Chad: [C: 032] Gerrit: Have a bogus valus for $gerrit_phab_cert [labs/private] - 10https://gerrit.wikimedia.org/r/341918 (owner: 10Paladox) [23:05:46] (03CR) 10Chad: [V: 032 C: 032] Gerrit: Have a bogus valus for $gerrit_phab_cert [labs/private] - 10https://gerrit.wikimedia.org/r/341918 (owner: 10Paladox) [23:06:28] (03CR) 10Paladox: "Need to press the submit thingy." [labs/private] - 10https://gerrit.wikimedia.org/r/341918 (owner: 10Paladox) [23:07:10] (03CR) 10Paladox: "Thanks :)" [labs/private] - 10https://gerrit.wikimedia.org/r/341918 (owner: 10Paladox) [23:37:45] (03Draft2) 10Zppix: Fix typo preventing phab notifications to ##Zppix-Wikipedia [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/341928 (https://phabricator.wikimedia.org/T159612) [23:39:25] (03CR) 10Zppix: [C: 031] Fix typo preventing phab notifications to ##Zppix-Wikipedia [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/341928 (https://phabricator.wikimedia.org/T159612) (owner: 10Zppix) [23:41:07] valhallasw`cloud: mind merging ^