[01:08:21] 11,05▄10,08▄12,12▄07,03▄03,09▄11,05▄06,05▄08,09▄04,02▄13,03▄11,12▄11,11▄08,04▄02,06▄06,09▄06 https://www.youtube.com/watch?v=--jYqzJErTI LRH IS LIVE NOW!! CALL 415-349-5666 #LRH EFNETzcckowbf: shinken-wm thcipriani|afk ebernhardson sir_lestaty Zppix Zppix|Away tgr|away nuria_ RoanKattouw Urbane jackmcbarn robh ema mdholloway TimStarling icinga-wm yurik [01:08:26] 03,02▄09,08▄13,10▄12,12▄08,04▄10,02▄03,09▄03,04▄08,04▄02,06▄12,11▄06 https://www.youtube.com/watch?v=--jYqzJErTI LRH IS LIVE NOW!! CALL 415-349-5666 #LRH EFNETqfgpjqcxri: sbailey Shanmugamp7 greg-g Zppix MGC ema robh brion thcipriani|afk Tobi_WMDE_SW_NA AaronSchulz Volker_E tgr|away marlier nuria_ icinga-wm FlorianSW ebernhardson TimStarling yurik ragesoss_ shinken-wm [01:08:31] 04,08▄12,03▄06,05▄04,13▄12,05▄07,05▄02,03▄11,05▄02,05▄05,03▄13,07▄02,06▄09,07▄11,02▄08,09▄02,13▄06,08▄11,03▄11,07▄11,06▄06 https://www.youtube.com/watch?v=--jYqzJErTI LRH IS LIVE NOW!! CALL 415-349-5666 #LRH EFNETkqwtg: shinken-wm gilles tgr|away Zppix nuria_ Urbane Zppix|Away jackmcbarn Tobi_WMDE_SW_NA TimStarling Shanmugamp7 ebern [01:08:36] 11,08▄08,10▄13,07▄09,04▄13,05▄10,03▄05,13▄12,09▄05,07▄05,13▄08,13▄09,06▄02,12▄05,06▄13,13▄13,02▄02,13▄04 https://www.youtube.com/watch?v=--jYqzJErTI LRH IS LIVE NOW!! CALL 415-349-5666 #LRH EFNETsbxel: addshore FlorianSW nuria_ marlier AaronSchulz TimStarling Zppix Tobi_WMDE_SW_NA ebernhardson eddiegp Shanmugamp7 godog brion sir_lestaty MGC jac [01:08:41] 02,11▄13,07▄05,05▄06,05▄06,10▄02,04▄07,10▄08,05▄04,12▄07,03▄10,05▄10,07▄02,07▄03,11▄06,05▄11,10▄09,06▄03,12▄09,07▄07,03▄09 https://www.youtube.com/watch?v=--jYqzJErTI LRH IS LIVE NOW!! CALL 415-349-5666 #LRH EFNETkyzqfp: quiddity brion yurik Reception123 sbailey mdholloway marlier RoanKattouw tgr|away FlorianSW godog Tobi_WMDE_SW_NA [01:08:46] 11,08▄06,02▄08,09▄10,12▄08,09▄12,12▄07,06▄13,12▄09,10▄03,10▄10,07▄03,05▄07,03▄06,12▄09,09▄10 https://www.youtube.com/watch?v=--jYqzJErTI LRH IS LIVE NOW!! CALL 415-349-5666 #LRH EFNETaivocg: marlier Reception123 RoanKattouw Shanmugamp7 fmla_dr0ptp4kt mdholloway Volker_E gilles ragesoss_ greg-g Krenair phedenskog ema sbailey TimStarling addshore Zppix|A [01:08:51] 11,11▄07,03▄02,06▄06,11▄13,02▄13,12▄12,06▄13,02▄10,03▄05,08▄08,11▄10,07▄08,05▄05,11▄12,10▄03,09▄10,05▄09,05▄11 https://www.youtube.com/watch?v=--jYqzJErTI LRH IS LIVE NOW!! CALL 415-349-5666 #LRH EFNETceyazcbahy: brion nuria_ eddiegp Zppix|Away greg-g AaronSchulz sbailey FlorianSW Krenair jackmcbarn RoanKattouw Urbane ema mdholloway TimStar [01:08:56] 05,05▄13,05▄02,10▄05,05▄06,03▄02,08▄11,12▄13,04▄02,08▄02,08▄08,06▄03,12▄12,09▄10,11▄05,03▄12,03▄13,02▄12,09▄09 https://www.youtube.com/watch?v=--jYqzJErTI LRH IS LIVE NOW!! CALL 415-349-5666 #LRH EFNETeoiwgfr: robh yurik brion phedenskog Reception123 Volker_E fmla_dr0ptp4kt eddiegp sbailey shinken-wm tgr|away gilles Tobi_WMDE_SW_NA marlier [01:09:01] 06,02▄10,12▄09,03▄10,10▄13,05▄02,13▄12,02▄07,08▄12,12▄12,09▄02,05▄07,07▄03 https://www.youtube.com/watch?v=--jYqzJErTI LRH IS LIVE NOW!! CALL 415-349-5666 #LRH EFNETnjkfgvaf: Shanmugamp7 jackmcbarn godog marlier Volker_E Krenair shinken-wm brion gilles RoanKattouw MGC Reception123 ragesoss_ icinga-wm sir_lestaty yurik ebernhardson FlorianSW AaronSchulz eddiegp add [01:09:06] 04,13▄06,11▄05,08▄02,04▄13,07▄05,13▄04,07▄07,11▄09,11▄12,10▄04,07▄10,02▄13,06▄09,04▄09 https://www.youtube.com/watch?v=--jYqzJErTI LRH IS LIVE NOW!! CALL 415-349-5666 #LRH EFNETzwgakjqu: marlier addshore MGC Volker_E gilles shinken-wm greg-g brion Reception123 Tobi_WMDE_SW_NA TimStarling godog sir_lestaty yurik Urbane Zppix jackmcbarn eddiegp mdholloway Aa [01:09:11] 12,02▄07,07▄13,02▄10,05▄02,09▄13,11▄13,06▄04,02▄13,08▄12,09▄03,05▄07 https://www.youtube.com/watch?v=--jYqzJErTI LRH IS LIVE NOW!! CALL 415-349-5666 #LRH EFNETowusyq: sir_lestaty MGC Urbane robh RoanKattouw Zppix|Away jackmcbarn Volker_E brion Tobi_WMDE_SW_NA Shanmugamp7 greg-g yurik fmla_dr0ptp4kt quiddity gilles ragesoss_ TimStarling AaronSchulz tgr|away eddiegp shi [01:09:16] 12,04▄05,10▄12,02▄11,06▄11,05▄06,10▄09,08▄02,06▄09,11▄07,12▄04 https://www.youtube.com/watch?v=--jYqzJErTI LRH IS LIVE NOW!! CALL 415-349-5666 #LRH EFNETlncexcore: Zppix|Away robh MGC sir_lestaty sbailey fmla_dr0ptp4kt mdholloway addshore Shanmugamp7 gilles Reception123 phedenskog thcipriani|afk Volker_E Zppix FlorianSW Krenair nuria_ AaronSchulz brion icinga-wm shinken-w [01:09:23] 12,13▄09,10▄02,07▄10,11▄10,13▄09,12▄04,05▄11,09▄03,04▄04,12▄02,03▄12,10▄08,10▄03,02▄02,13▄03,11▄10,10▄09 https://www.youtube.com/watch?v=--jYqzJErTI LRH IS LIVE NOW!! CALL 415-349-5666 #LRH EFNETtjrmwfjq: MGC Zppix|Away shinken-wm gilles Volker_E ragesoss_ tgr|away jackmcbarn RoanKattouw sir_lestaty Tobi_WMDE_SW_NA mdholloway greg-g nuria_ Kren [01:09:26] 11,05▄06,09▄02,06▄07,02▄10,09▄04,09▄05,03▄06,07▄12,13▄10,08▄10,09▄03,13▄03,10▄08 https://www.youtube.com/watch?v=--jYqzJErTI LRH IS LIVE NOW!! CALL 415-349-5666 #LRH EFNETypsrjea: fmla_dr0ptp4kt RoanKattouw nuria_ FlorianSW Urbane AaronSchulz yurik quiddity Tobi_WMDE_SW_NA Reception123 marlier shinken-wm eddiegp Zppix MGC ragesoss_ thcipriani|afk Krenair addsh [01:09:31] 05,04▄12,03▄02,07▄05,03▄13,04▄06,13▄06,05▄08,13▄06,02▄13,12▄02,13▄03,06▄11,13▄02,10▄09,10▄05,12▄04,06▄06,13▄12 https://www.youtube.com/watch?v=--jYqzJErTI LRH IS LIVE NOW!! CALL 415-349-5666 #LRH EFNETjhqavcwyrb: Krenair eddiegp nuria_ AaronSchulz ragesoss_ FlorianSW RoanKattouw Reception123 addshore Shanmugamp7 greg-g marlier sbailey gille [01:09:36] 09,06▄09,06▄05,12▄06,04▄13,07▄08,11▄09,05▄08,02▄06,08▄11,11▄11,07▄13 https://www.youtube.com/watch?v=--jYqzJErTI LRH IS LIVE NOW!! CALL 415-349-5666 #LRH EFNETfpumkzl: Volker_E tgr|away Krenair AaronSchulz Tobi_WMDE_SW_NA Reception123 shinken-wm sbailey mdholloway gilles phedenskog thcipriani|afk nuria_ yurik robh Urbane addshore ebernhardson ragesoss_ fmla_dr0ptp4kt [01:09:41] 09,04▄05,02▄10,10▄07,11▄11,10▄11,10▄06,13▄05,11▄05,08▄02,12▄02,06▄05,09▄10,08▄06,11▄06 https://www.youtube.com/watch?v=--jYqzJErTI LRH IS LIVE NOW!! CALL 415-349-5666 #LRH EFNETjtvlmqdhh: Shanmugamp7 shinken-wm ebernhardson jackmcbarn brion TimStarling gilles yurik robh marlier Tobi_WMDE_SW_NA nuria_ Reception123 FlorianSW AaronSchulz sir_lestaty Krenair U [01:09:46] 11,11▄11,03▄09,05▄05,09▄07,11▄07,10▄05,11▄09,06▄07,07▄08,02▄11,08▄04,03▄12,05▄02,03▄05,09▄12,13▄04,06▄04,03▄05,09▄10,12▄04 https://www.youtube.com/watch?v=--jYqzJErTI LRH IS LIVE NOW!! CALL 415-349-5666 #LRH EFNETjhymbljjv: eddiegp Reception123 thcipriani|afk mdholloway fmla_dr0ptp4kt RoanKattouw godog sbailey ragesoss_ Krenair adds [01:09:51] 13,09▄09,08▄10,12▄13,03▄07,08▄02,02▄06,02▄02,10▄05,08▄10,09▄05,11▄04,12▄10 https://www.youtube.com/watch?v=--jYqzJErTI LRH IS LIVE NOW!! CALL 415-349-5666 #LRH EFNETcmivabd: sir_lestaty Reception123 yurik robh ragesoss_ nuria_ gilles Krenair AaronSchulz Tobi_WMDE_SW_NA FlorianSW MGC Zppix|Away brion mdholloway ema fmla_dr0ptp4kt RoanKattouw godog tgr|away shinken- [01:09:56] 08,07▄10,02▄10,13▄10,12▄11,02▄11,02▄07,04▄07,11▄09,10▄08,13▄12,11▄11,12▄02,08▄13,05▄12,08▄07,06▄04,06▄03 https://www.youtube.com/watch?v=--jYqzJErTI LRH IS LIVE NOW!! CALL 415-349-5666 #LRH EFNETfraauifj: fmla_dr0ptp4kt FlorianSW sir_lestaty Volker_E godog Zppix ema marlier sbailey MGC mdholloway Urbane addshore ebernhardson brion gilles Recept [01:10:01] 10,10▄06,06▄07,06▄05,08▄06,09▄05,10▄10,12▄03,03▄07,10▄09,10▄11,10▄07,13▄03,07▄13,02▄12,02▄13,02▄12,10▄10,13▄06,13▄13 https://www.youtube.com/watch?v=--jYqzJErTI LRH IS LIVE NOW!! CALL 415-349-5666 #LRH EFNEThoamv: fmla_dr0ptp4kt MGC Volker_E mdholloway ebernhardson Reception123 quiddity greg-g Zppix|Away thcipriani|afk robh Zppix AaronS [01:10:05] PROBLEM - Puppet errors on deployment-sca01 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [01:10:06] 09,09▄06,09▄13,03▄06,07▄13,09▄09,05▄05,05▄10,02▄04,02▄02,04▄08,02▄08,09▄02,12▄03,09▄05,03▄09,06▄13,06▄06,05▄12,13▄11,05▄03 https://www.youtube.com/watch?v=--jYqzJErTI LRH IS LIVE NOW!! CALL 415-349-5666 #LRH EFNEThbeydwepj: ebernhardson greg-g sbailey gilles FlorianSW marlier robh yurik mdholloway nuria_ ragesoss_ Tobi_WMDE_SW_NA Ti [01:10:11] 03,02▄08,12▄04,05▄12,12▄12,06▄07,02▄03,03▄08,06▄13,12▄12,13▄10,12▄07,10▄12,11▄04,02▄13,12▄07 https://www.youtube.com/watch?v=--jYqzJErTI LRH IS LIVE NOW!! CALL 415-349-5666 #LRH EFNETpfjnl: greg-g TimStarling sir_lestaty Shanmugamp7 phedenskog sbailey ebernhardson thcipriani|afk marlier Zppix jackmcbarn ragesoss_ RoanKattouw fmla_dr0ptp4kt ema AaronSch [01:10:17] 10,02▄07,07▄03,13▄08,06▄08,10▄08,02▄13,09▄10,11▄13,03▄06,11▄13,11▄09,13▄06,07▄13,07▄09,06▄10,05▄13,13▄03,12▄03 https://www.youtube.com/watch?v=--jYqzJErTI LRH IS LIVE NOW!! CALL 415-349-5666 #LRH EFNETpttcndjts: FlorianSW yurik TimStarling quiddity sir_lestaty icinga-wm thcipriani|afk greg-g fmla_dr0ptp4kt ebernhardson Volker_E Zppix shinke [01:10:21] 07,10▄02,04▄08,10▄08,10▄07,04▄06,09▄10,03▄13,10▄07,13▄11,07▄12 https://www.youtube.com/watch?v=--jYqzJErTI LRH IS LIVE NOW!! CALL 415-349-5666 #LRH EFNETyqwsefd: gilles nuria_ icinga-wm eddiegp mdholloway Urbane shinken-wm Zppix ema brion ebernhardson AaronSchulz Zppix|Away RoanKattouw Tobi_WMDE_SW_NA Reception123 Krenair Shanmugamp7 greg-g robh TimStarling sbailey sir_le [01:10:26] 10,03▄08,08▄11,06▄13,10▄11,05▄10,10▄13,05▄03,02▄04,07▄04,12▄11 https://www.youtube.com/watch?v=--jYqzJErTI LRH IS LIVE NOW!! CALL 415-349-5666 #LRH EFNETbowptqa: jackmcbarn addshore greg-g marlier eddiegp Zppix nuria_ yurik ebernhardson mdholloway godog MGC RoanKattouw Tobi_WMDE_SW_NA ragesoss_ Reception123 phedenskog brion shinken-wm AaronSchulz icinga-wm Urbane gilles q [01:10:31] !ops [01:10:31] 13,04▄12,11▄02,13▄13,10▄06,06▄03,03▄05,07▄09,08▄10,12▄03,09▄06,08▄11,05▄06 https://www.youtube.com/watch?v=--jYqzJErTI LRH IS LIVE NOW!! CALL 415-349-5666 #LRH EFNETfyjwg: sir_lestaty Tobi_WMDE_SW_NA Reception123 MGC addshore AaronSchulz jackmcbarn shinken-wm fmla_dr0ptp4kt FlorianSW nuria_ Urbane Zppix icinga-wm Krenair yurik RoanKattouw thcipriani|afk quiddity e [01:10:36] 04,03▄07,12▄06,08▄11,05▄11,13▄13,02▄13,08▄12,07▄09,12▄10,04▄13,11▄07,10▄05,04▄02,13▄09 https://www.youtube.com/watch?v=--jYqzJErTI LRH IS LIVE NOW!! CALL 415-349-5666 #LRH EFNETehyhpsbbt: Shanmugamp7 ema Zppix|Away brion marlier Krenair eddiegp jackmcbarn Reception123 mdholloway greg-g FlorianSW MGC yurik fmla_dr0ptp4kt Tobi_WMDE_SW_NA tgr|away thcipriani| [01:10:41] 11,06▄03,02▄07,09▄09,08▄05,03▄11,05▄12,09▄11,13▄11,02▄09,13▄09,12▄02,12▄09,04▄12,13▄02,08▄07,09▄11 https://www.youtube.com/watch?v=--jYqzJErTI LRH IS LIVE NOW!! CALL 415-349-5666 #LRH EFNETexnkug: Krenair jackmcbarn Reception123 eddiegp robh ebernhardson Tobi_WMDE_SW_NA ema icinga-wm godog tgr|away ragesoss_ thcipriani|afk sbailey yurik fmla_dr0ptp [01:10:46] 08,07▄07,09▄07,08▄05,07▄08,10▄13,13▄02,04▄08,07▄09,11▄10,02▄04,10▄03,11▄12,12▄13,13▄11,08▄02,10▄11,11▄09 https://www.youtube.com/watch?v=--jYqzJErTI LRH IS LIVE NOW!! CALL 415-349-5666 #LRH EFNETbxwnr: MGC yurik fmla_dr0ptp4kt sbailey Tobi_WMDE_SW_NA ebernhardson marlier ragesoss_ phedenskog quiddity jackmcbarn Zppix|Away thcipriani|afk icinga- [01:10:52] 03,05▄10,04▄12,11▄13,05▄08,12▄11,05▄06,10▄12,08▄12,06▄04,08▄08,09▄03,13▄13,02▄09,06▄10,04▄10,10▄06 https://www.youtube.com/watch?v=--jYqzJErTI LRH IS LIVE NOW!! CALL 415-349-5666 #LRH EFNETurzqux: phedenskog thcipriani|afk Tobi_WMDE_SW_NA ragesoss_ Zppix FlorianSW MGC fmla_dr0ptp4kt Shanmugamp7 RoanKattouw robh ema AaronSchulz ebernhardson Zppix|Aw [01:10:56] 07,09▄02,05▄09,03▄08,11▄13,04▄03,11▄10,09▄09,12▄03,03▄08,10▄07,11▄08,02▄03,09▄10,13▄10,07▄02,12▄07,12▄05,02▄02,09▄08 https://www.youtube.com/watch?v=--jYqzJErTI LRH IS LIVE NOW!! CALL 415-349-5666 #LRH EFNETyplfno: phedenskog addshore yurik Krenair shinken-wm sbailey Volker_E godog Reception123 Tobi_WMDE_SW_NA thcipriani|afk quiddity Zp [01:11:01] 07,08▄11,09▄13,12▄05,07▄13,09▄03,13▄04,09▄08,02▄09,03▄03,04▄12,07▄07,07▄11,04▄13,10▄09 https://www.youtube.com/watch?v=--jYqzJErTI LRH IS LIVE NOW!! CALL 415-349-5666 #LRH EFNETsmpim: gilles mdholloway phedenskog thcipriani|afk Krenair eddiegp TimStarling Reception123 brion ema addshore MGC RoanKattouw fmla_dr0ptp4kt shinken-wm yurik Tobi_WMDE_SW_NA tgr|aw [01:11:07] 06,03▄10,06▄10,07▄05,06▄12,03▄10,03▄12,09▄06,09▄02,06▄10,08▄03,05▄09,04▄05,12▄02,08▄10,12▄07,06▄08,10▄04 https://www.youtube.com/watch?v=--jYqzJErTI LRH IS LIVE NOW!! CALL 415-349-5666 #LRH EFNETilerjhfimn: mdholloway RoanKattouw ebernhardson thcipriani|afk sbailey tgr|away godog robh quiddity icinga-wm FlorianSW Shanmugamp7 eddiegp yurik rages [01:11:11] 05,06▄08,11▄11,08▄11,13▄07,03▄06,13▄10,12▄04,11▄08,11▄10,08▄07,07▄03,13▄07 https://www.youtube.com/watch?v=--jYqzJErTI LRH IS LIVE NOW!! CALL 415-349-5666 #LRH EFNETrqvical: ema nuria_ shinken-wm FlorianSW TimStarling RoanKattouw Krenair brion robh MGC mdholloway tgr|away Zppix|Away yurik eddiegp ragesoss_ jackmcbarn phedenskog Urbane godog fmla_dr0ptp4kt thcipria [01:11:16] 06,13▄12,06▄07,05▄12,04▄05,09▄06,05▄04,12▄12,03▄08,08▄04,10▄05,10▄02,07▄06,03▄12,05▄07 https://www.youtube.com/watch?v=--jYqzJErTI LRH IS LIVE NOW!! CALL 415-349-5666 #LRH EFNETpbdrkng: shinken-wm MGC quiddity Volker_E icinga-wm thcipriani|afk addshore fmla_dr0ptp4kt tgr|away Shanmugamp7 ragesoss_ Zppix|Away brion mdholloway TimStarling AaronSchulz marlier [01:11:21] 09,06▄02,11▄08,04▄10,09▄11,03▄06,02▄10,07▄04,03▄12,13▄07,05▄13,07▄13,03▄07,13▄04,05▄08,06▄03,04▄11,11▄06,13▄05,05▄13 https://www.youtube.com/watch?v=--jYqzJErTI LRH IS LIVE NOW!! CALL 415-349-5666 #LRH EFNETynmlcu: tgr|away gilles fmla_dr0ptp4kt yurik ebernhardson TimStarling thcipriani|afk icinga-wm FlorianSW ragesoss_ Shanmugamp7 Zppi [01:11:26] 03,02▄08,02▄10,07▄08,11▄04,07▄11,02▄09,13▄06,03▄04,05▄13,05▄09,12▄05,13▄02,07▄03,06▄03,03▄06,02▄11,07▄08 https://www.youtube.com/watch?v=--jYqzJErTI LRH IS LIVE NOW!! CALL 415-349-5666 #LRH EFNETqwljbnn: tgr|away mdholloway sbailey MGC Zppix shinken-wm RoanKattouw ebernhardson jackmcbarn marlier thcipriani|afk nuria_ Krenair quiddity gilles ici [01:11:31] 12,10▄12,02▄13,03▄04,05▄11,12▄05,10▄13,08▄10,04▄07,10▄08,11▄13 https://www.youtube.com/watch?v=--jYqzJErTI LRH IS LIVE NOW!! CALL 415-349-5666 #LRH EFNETibmweuvvjc: mdholloway godog RoanKattouw marlier TimStarling Reception123 nuria_ Shanmugamp7 thcipriani|afk jackmcbarn sir_lestaty addshore ragesoss_ Urbane phedenskog Volker_E sbailey yurik tgr|away FlorianSW ema fmla_dr [01:11:36] 12,13▄02,07▄04,09▄02,10▄02,04▄13,06▄06,04▄03,12▄08,02▄10,12▄13,04▄10,05▄08 https://www.youtube.com/watch?v=--jYqzJErTI LRH IS LIVE NOW!! CALL 415-349-5666 #LRH EFNETyerlspm: nuria_ Tobi_WMDE_SW_NA icinga-wm Zppix|Away eddiegp gilles marlier AaronSchulz jackmcbarn phedenskog quiddity Reception123 mdholloway greg-g addshore Krenair sbailey shinken-wm TimStarling bri [01:11:41] 12,03▄02,10▄13,11▄13,08▄07,09▄11,04▄12,03▄07,05▄11,03▄06,07▄12,04▄02 https://www.youtube.com/watch?v=--jYqzJErTI LRH IS LIVE NOW!! CALL 415-349-5666 #LRH EFNETpdejoepgu: ebernhardson jackmcbarn TimStarling AaronSchulz greg-g shinken-wm FlorianSW Zppix nuria_ ragesoss_ Zppix|Away quiddity sir_lestaty gilles sbailey yurik mdholloway marlier thcipriani|afk RoanKattouw fm [01:11:46] 12,08▄04,07▄12,03▄05,04▄04,10▄06,03▄12,04▄04,03▄03,13▄11,12▄07,12▄09,09▄07,05▄10 https://www.youtube.com/watch?v=--jYqzJErTI LRH IS LIVE NOW!! CALL 415-349-5666 #LRH EFNETihzcd: ragesoss_ Shanmugamp7 ebernhardson Zppix|Away Volker_E thcipriani|afk brion icinga-wm FlorianSW eddiegp AaronSchulz gilles godog sir_lestaty mdholloway sbailey quiddity RoanKattouw Urb [01:11:51] 03,13▄13,04▄10,13▄05,03▄13,10▄04,07▄10,13▄03,11▄13,12▄08,05▄09,08▄04,13▄12,03▄12,10▄05,06▄08 https://www.youtube.com/watch?v=--jYqzJErTI LRH IS LIVE NOW!! CALL 415-349-5666 #LRH EFNETohpqdqgq: ema AaronSchulz Volker_E ragesoss_ ebernhardson marlier Zppix|Away Reception123 icinga-wm Zppix Krenair jackmcbarn thcipriani|afk eddiegp gilles RoanKattouw fmla [01:11:56] 04,05▄04,04▄11,10▄07,13▄04,10▄11,09▄06,11▄12,10▄12,12▄12,12▄09,09▄02,04▄12,12▄11,12▄08,03▄04 https://www.youtube.com/watch?v=--jYqzJErTI LRH IS LIVE NOW!! CALL 415-349-5666 #LRH EFNETecgmd: Shanmugamp7 addshore shinken-wm gilles robh Volker_E quiddity greg-g godog marlier sbailey ema FlorianSW RoanKattouw jackmcbarn icinga-wm yurik MGC Krenair Zppix|Aw [01:12:03] 08,04▄12,08▄03,13▄04,09▄13,02▄03,02▄10,12▄05,08▄12,06▄12,05▄02,06▄06,03▄09,08▄13,06▄05,12▄05,05▄04,04▄12,04▄09,08▄04 https://www.youtube.com/watch?v=--jYqzJErTI LRH IS LIVE NOW!! CALL 415-349-5666 #LRH EFNETbdgkxs: TimStarling RoanKattouw gilles robh Volker_E nuria_ thcipriani|afk FlorianSW AaronSchulz tgr|away Urbane eddiegp brion phed [01:12:06] 10,07▄06,13▄10,13▄09,06▄02,12▄12,12▄11,13▄03,08▄06,07▄05,10▄03,02▄08,12▄10,11▄07,09▄07 https://www.youtube.com/watch?v=--jYqzJErTI LRH IS LIVE NOW!! CALL 415-349-5666 #LRH EFNETcxphn: shinken-wm robh Reception123 FlorianSW gilles MGC Urbane phedenskog ebernhardson addshore jackmcbarn sir_lestaty quiddity greg-g icinga-wm fmla_dr0ptp4kt mdholloway RoanKatto [01:12:11] 09,05▄05,05▄07,13▄04,10▄09,10▄11,09▄11,13▄04,09▄12,07▄06,06▄09,13▄06 https://www.youtube.com/watch?v=--jYqzJErTI LRH IS LIVE NOW!! CALL 415-349-5666 #LRH EFNETgotwordbi: shinken-wm gilles eddiegp MGC brion RoanKattouw sbailey Zppix|Away quiddity TimStarling addshore nuria_ ebernhardson godog Tobi_WMDE_SW_NA Krenair greg-g phedenskog robh AaronSchulz ragesoss_ FlorianS [01:12:16] 13,02▄11,09▄02,06▄05,03▄13,05▄07,10▄06,12▄06,08▄03,09▄06,08▄13,06▄04,02▄03,11▄03,10▄05,05▄12,07▄10,13▄03 https://www.youtube.com/watch?v=--jYqzJErTI LRH IS LIVE NOW!! CALL 415-349-5666 #LRH EFNETktyzfuttu: robh greg-g quiddity gilles jackmcbarn Krenair godog ragesoss_ Volker_E icinga-wm phedenskog addshore MGC sbailey sir_lestaty mdholloway ebe [01:12:21] 10,13▄02,10▄12,10▄09,06▄11,09▄10,09▄04,13▄03,12▄06,07▄02,07▄08,13▄11,07▄09 https://www.youtube.com/watch?v=--jYqzJErTI LRH IS LIVE NOW!! CALL 415-349-5666 #LRH EFNEThgsivxa: nuria_ FlorianSW ema addshore sir_lestaty Zppix ebernhardson Urbane marlier ragesoss_ phedenskog Tobi_WMDE_SW_NA shinken-wm AaronSchulz Volker_E Shanmugamp7 robh tgr|away brion yurik Reception [01:12:26] 10,06▄05,03▄06,06▄04,10▄12,08▄10,04▄05,05▄09,04▄13,10▄10,08▄07,12▄11,03▄03,05▄10,11▄13,11▄08,09▄06,09▄11 https://www.youtube.com/watch?v=--jYqzJErTI LRH IS LIVE NOW!! CALL 415-349-5666 #LRH EFNETlrefcr: AaronSchulz RoanKattouw addshore ebernhardson brion quiddity MGC yurik Tobi_WMDE_SW_NA Shanmugamp7 greg-g Zppix|Away gilles Reception123 sir_le [01:12:33] 13,05▄10,06▄04,08▄03,10▄05,11▄11,02▄06,05▄05,11▄06,03▄09,03▄02,05▄12,10▄11 https://www.youtube.com/watch?v=--jYqzJErTI LRH IS LIVE NOW!! CALL 415-349-5666 #LRH EFNETvlhtfxgqh: mdholloway greg-g quiddity Tobi_WMDE_SW_NA TimStarling icinga-wm marlier brion Reception123 Zppix|Away phedenskog gilles sir_lestaty MGC FlorianSW RoanKattouw thcipriani|afk addshore fmla_dr [01:12:36] 03,02▄02,08▄09,10▄12,08▄06,03▄07,08▄10,08▄06,05▄02,09▄11,13▄02,05▄06,05▄02 https://www.youtube.com/watch?v=--jYqzJErTI LRH IS LIVE NOW!! CALL 415-349-5666 #LRH EFNETfkwkhfrqo: addshore Urbane ema robh yurik TimStarling ragesoss_ mdholloway gilles Tobi_WMDE_SW_NA Zppix|Away godog Zppix AaronSchulz marlier eddiegp Volker_E shinken-wm sir_lestaty icinga-wm quiddity s [01:12:41] 08,09▄12,05▄02,05▄11,05▄08,04▄09,06▄08,10▄06,06▄10,08▄11,12▄11,11▄02,08▄11 https://www.youtube.com/watch?v=--jYqzJErTI LRH IS LIVE NOW!! CALL 415-349-5666 #LRH EFNETjfvvpq: icinga-wm shinken-wm Volker_E brion Urbane sbailey greg-g tgr|away AaronSchulz Reception123 Zppix|Away TimStarling mdholloway FlorianSW ragesoss_ robh godog quiddity addshore sir_lestaty gilles [01:12:46] 11,11▄10,05▄12,09▄02,10▄07,07▄03,09▄04,08▄05,04▄08,04▄09,06▄09,10▄07,02▄02,03▄10,03▄08,12▄03,07▄11,06▄03,03▄11,10▄10 https://www.youtube.com/watch?v=--jYqzJErTI LRH IS LIVE NOW!! CALL 415-349-5666 #LRH EFNEToqaofd: brion godog AaronSchulz Volker_E Zppix|Away addshore jackmcbarn Reception123 ebernhardson icinga-wm fmla_dr0ptp4kt mdhollow [01:12:51] 03,12▄13,06▄13,02▄03,04▄11,10▄13,05▄06,07▄03,06▄13,12▄02,08▄10,11▄07,09▄05,05▄02,02▄13,07▄09,02▄06,07▄03,07▄13,03▄11,13▄10 https://www.youtube.com/watch?v=--jYqzJErTI LRH IS LIVE NOW!! CALL 415-349-5666 #LRH EFNETglwrqbjcqv: sbailey RoanKattouw Krenair yurik gilles ema marlier Zppix|Away icinga-wm addshore Zppix ragesoss_ MGC sir_le [01:12:57] 04,02▄11,13▄02,09▄07,12▄09,07▄03,11▄10,11▄09,12▄07,02▄05,12▄12 https://www.youtube.com/watch?v=--jYqzJErTI LRH IS LIVE NOW!! CALL 415-349-5666 #LRH EFNETnkfjz: eddiegp Zppix|Away RoanKattouw Zppix FlorianSW jackmcbarn Tobi_WMDE_SW_NA Krenair addshore tgr|away quiddity phedenskog sbailey gilles robh mdholloway TimStarling brion greg-g yurik thcipriani|afk ragesoss_ sir_les [01:13:02] 02,07▄09,11▄12,13▄06,07▄13,07▄06,13▄08,02▄10,12▄04,05▄07,02▄12,04▄09,10▄06 https://www.youtube.com/watch?v=--jYqzJErTI LRH IS LIVE NOW!! CALL 415-349-5666 #LRH EFNEThwmldvtwhc: Shanmugamp7 robh Zppix|Away greg-g FlorianSW ebernhardson yurik brion Krenair sir_lestaty Reception123 fmla_dr0ptp4kt eddiegp ragesoss_ RoanKattouw icinga-wm addshore MGC Tobi_WMDE_SW_NA qu [01:13:06] 13,06▄04,05▄07,13▄04,07▄03,04▄05,09▄06,12▄06,11▄07,11▄11,06▄06,07▄10,03▄04,10▄09,13▄07,08▄05,05▄04,10▄09 https://www.youtube.com/watch?v=--jYqzJErTI LRH IS LIVE NOW!! CALL 415-349-5666 #LRH EFNETdifwejjajs: sbailey godog jackmcbarn Zppix|Away marlier greg-g FlorianSW tgr|away AaronSchulz Zppix Shanmugamp7 Reception123 sir_lestaty phedenskog yur [01:13:11] 10,08▄11,11▄13,12▄06,04▄12,12▄02,02▄10,05▄04,10▄09,04▄07,08▄08,13▄05,06▄11,07▄02,06▄03,02▄09,13▄04,10▄12,13▄04,05▄06 https://www.youtube.com/watch?v=--jYqzJErTI LRH IS LIVE NOW!! CALL 415-349-5666 #LRH EFNETizvsxvvpis: Shanmugamp7 gilles Tobi_WMDE_SW_NA FlorianSW Volker_E sir_lestaty godog brion Krenair nuria_ sbailey robh Reception123 [01:13:17] 12,09▄08,09▄08,06▄06,09▄07,06▄11,03▄09,11▄03,08▄02,03▄11,05▄08,07▄12,04▄11,10▄07,12▄08,09▄12 https://www.youtube.com/watch?v=--jYqzJErTI LRH IS LIVE NOW!! CALL 415-349-5666 #LRH EFNETddubey: Volker_E yurik ebernhardson greg-g fmla_dr0ptp4kt nuria_ Tobi_WMDE_SW_NA Krenair FlorianSW Reception123 brion godog thcipriani|afk icinga-wm tgr|away MGC jackmcbar [01:13:22] 06,03▄05,03▄06,03▄06,07▄12,10▄07,06▄02,09▄02,05▄11,08▄06,03▄10 https://www.youtube.com/watch?v=--jYqzJErTI LRH IS LIVE NOW!! CALL 415-349-5666 #LRH EFNETsvfcr: icinga-wm tgr|away gilles Urbane robh Reception123 Zppix|Away jackmcbarn phedenskog Zppix godog marlier shinken-wm addshore ebernhardson brion MGC mdholloway nuria_ Tobi_WMDE_SW_NA Krenair sbailey thcipriani|afk Ti [01:13:27] 03,13▄02,03▄12,13▄13,07▄11,02▄06,13▄04,12▄12,04▄04,05▄11,03▄07,04▄10,11▄09,13▄07,08▄13,11▄08,09▄05,13▄04 https://www.youtube.com/watch?v=--jYqzJErTI LRH IS LIVE NOW!! CALL 415-349-5666 #LRH EFNETlyrej: robh Zppix TimStarling yurik MGC godog Volker_E gilles thcipriani|afk tgr|away ragesoss_ icinga-wm mdholloway phedenskog Shanmugamp7 jackmcbarn [01:13:31] 07,11▄13,06▄02,07▄13,03▄06,09▄10,10▄04,12▄09,03▄06,10▄04,08▄02,07▄10,05▄12,06▄10 https://www.youtube.com/watch?v=--jYqzJErTI LRH IS LIVE NOW!! CALL 415-349-5666 #LRH EFNETictpyzfe: nuria_ Zppix icinga-wm quiddity ebernhardson Shanmugamp7 Volker_E FlorianSW addshore Zppix|Away Tobi_WMDE_SW_NA fmla_dr0ptp4kt gilles greg-g sir_lestaty marlier RoanKattouw MGC Rece [01:13:36] 03,09▄13,13▄07,05▄02,05▄03,08▄06,06▄05,13▄07,12▄05,13▄07,10▄07,10▄12,12▄08,02▄06,08▄10,09▄05,04▄02,13▄04,05▄10,06▄13 https://www.youtube.com/watch?v=--jYqzJErTI LRH IS LIVE NOW!! CALL 415-349-5666 #LRH EFNETholxdhouje: sbailey nuria_ Volker_E icinga-wm Zppix shinken-wm addshore brion gilles RoanKattouw Shanmugamp7 Tobi_WMDE_SW_NA godog [01:13:41] 10,10▄03,04▄02,08▄06,09▄13,04▄09,12▄13,12▄11,05▄09,11▄09,02▄07,10▄02,05▄06,09▄03 https://www.youtube.com/watch?v=--jYqzJErTI LRH IS LIVE NOW!! CALL 415-349-5666 #LRH EFNETjoenrqq: Zppix Tobi_WMDE_SW_NA godog robh tgr|away icinga-wm Krenair eddiegp brion addshore nuria_ Urbane fmla_dr0ptp4kt sbailey RoanKattouw quiddity ragesoss_ marlier thcipriani|afk ebernhar [01:13:46] 07,09▄07,07▄02,03▄13,03▄11,07▄02,07▄03,09▄11,11▄05,06▄06,12▄03,08▄08 https://www.youtube.com/watch?v=--jYqzJErTI LRH IS LIVE NOW!! CALL 415-349-5666 #LRH EFNETbrhuuzzcta: TimStarling yurik sir_lestaty ragesoss_ Krenair sbailey jackmcbarn addshore eddiegp quiddity RoanKattouw fmla_dr0ptp4kt nuria_ ebernhardson Tobi_WMDE_SW_NA tgr|away Volker_E icinga-wm marlier Zppix Z [01:13:52] 12,05▄06,09▄06,08▄13,11▄09,13▄05,02▄05,12▄08,11▄04,13▄11,02▄11,10▄07,13▄12,10▄13,04▄03,12▄04,11▄03,13▄04,04▄02 https://www.youtube.com/watch?v=--jYqzJErTI LRH IS LIVE NOW!! CALL 415-349-5666 #LRH EFNETptcvevkg: Shanmugamp7 Krenair quiddity Reception123 Volker_E MGC RoanKattouw Tobi_WMDE_SW_NA yurik sir_lestaty ebernhardson icinga-wm tgr|awa [01:13:56] 08,11▄02,06▄09,11▄06,08▄08,06▄07,09▄06,04▄06,09▄06,08▄08,08▄11,10▄03,06▄12,03▄02,09▄12,13▄08,02▄13 https://www.youtube.com/watch?v=--jYqzJErTI LRH IS LIVE NOW!! CALL 415-349-5666 #LRH EFNETvireapgbi: mdholloway greg-g thcipriani|afk ema addshore jackmcbarn Volker_E Zppix|Away eddiegp FlorianSW Reception123 Zppix icinga-wm shinken-wm robh tgr|away n [01:14:01] 08,04▄03,11▄07,08▄04,09▄04,08▄05,12▄06,09▄05,06▄07,08▄08,04▄12,11▄12,04▄10,02▄12,08▄11,09▄06,08▄07 https://www.youtube.com/watch?v=--jYqzJErTI LRH IS LIVE NOW!! CALL 415-349-5666 #LRH EFNETaikotehf: AaronSchulz Tobi_WMDE_SW_NA robh Urbane phedenskog nuria_ yurik Zppix|Away TimStarling ebernhardson MGC Volker_E eddiegp FlorianSW quiddity sbailey god [01:14:07] 07,13▄11,05▄03,13▄11,02▄07,07▄12,08▄03,09▄08,05▄03,08▄03,03▄13,03▄09,08▄04,09▄09,06▄13 https://www.youtube.com/watch?v=--jYqzJErTI LRH IS LIVE NOW!! CALL 415-349-5666 #LRH EFNETdgxqyst: AaronSchulz tgr|away TimStarling greg-g Zppix|Away yurik eddiegp ema ebernhardson Zppix brion Reception123 sir_lestaty shinken-wm MGC sbailey quiddity gilles icinga-wm adds [01:14:11] 06,12▄04,02▄06,02▄08,11▄03,03▄08,07▄04,06▄09,12▄04,12▄10,08▄11,02▄09,13▄04,12▄06,12▄05,02▄02,02▄13 https://www.youtube.com/watch?v=--jYqzJErTI LRH IS LIVE NOW!! CALL 415-349-5666 #LRH EFNETwmilmqq: sir_lestaty ebernhardson Zppix Zppix|Away sbailey Urbane godog gilles yurik Reception123 Volker_E nuria_ AaronSchulz Tobi_WMDE_SW_NA quiddity thcipriani [01:14:17] 03,08▄12,02▄11,13▄07,13▄12,07▄11,10▄04,04▄05,11▄07,13▄05,10▄06,06▄10,04▄10,09▄09,12▄08,13▄07,12▄02,07▄07,05▄07,08▄08,03▄08 https://www.youtube.com/watch?v=--jYqzJErTI LRH IS LIVE NOW!! CALL 415-349-5666 #LRH EFNETrutiylanne: phedenskog marlier Zppix godog brion jackmcbarn ema greg-g Zppix|Away TimStarling ebernhardson robh mdhollowa [01:14:17] !ops [01:14:22] 07,07▄13,06▄04,06▄07,10▄12,10▄06,07▄10,09▄04,03▄05,03▄11,07▄04,09▄02 https://www.youtube.com/watch?v=--jYqzJErTI LRH IS LIVE NOW!! CALL 415-349-5666 #LRH EFNETkczzgdgirl: nuria_ addshore Zppix Shanmugamp7 quiddity marlier Krenair ragesoss_ Volker_E Reception123 Tobi_WMDE_SW_NA icinga-wm greg-g ebernhardson TimStarling tgr|away yurik robh Zppix|Away sir_lestaty thcipri [01:14:27] 06,07▄11,06▄10,02▄09,09▄02,03▄10,04▄09,13▄03,11▄07,04▄06,12▄13,10▄04 https://www.youtube.com/watch?v=--jYqzJErTI LRH IS LIVE NOW!! CALL 415-349-5666 #LRH EFNETphqufard: Urbane marlier ragesoss_ ema jackmcbarn Reception123 eddiegp Zppix Zppix|Away Krenair Volker_E Shanmugamp7 sir_lestaty FlorianSW TimStarling AaronSchulz shinken-wm fmla_dr0ptp4kt robh addshore gilles b [01:14:32] 03,10▄08,08▄11,02▄07,06▄12,05▄09,03▄09,09▄05,02▄08,02▄02,13▄12,05▄04,07▄02,12▄12,03▄10,13▄04,09▄11,03▄04,07▄07,10▄08,04▄02 https://www.youtube.com/watch?v=--jYqzJErTI LRH IS LIVE NOW!! CALL 415-349-5666 #LRH EFNETucailv: sbailey jackmcbarn ema Reception123 nuria_ FlorianSW thcipriani|afk Krenair godog quiddity mdholloway greg-g eber [01:14:36] 08,08▄10,04▄05,03▄06,13▄03,05▄07,11▄11,11▄11,13▄06,06▄11,04▄10,09▄03,06▄03,08▄11,13▄11 https://www.youtube.com/watch?v=--jYqzJErTI LRH IS LIVE NOW!! CALL 415-349-5666 #LRH EFNETtlvmq: gilles RoanKattouw FlorianSW Krenair tgr|away shinken-wm phedenskog addshore mdholloway nuria_ godog Zppix quiddity jackmcbarn brion TimStarling sbailey Urbane Reception123 Z [01:14:42] 05,12▄09,07▄03,09▄10,02▄06,12▄09,12▄03,13▄05,13▄09,13▄04,02▄11,12▄13,12▄09,03▄06,12▄12,10▄02 https://www.youtube.com/watch?v=--jYqzJErTI LRH IS LIVE NOW!! CALL 415-349-5666 #LRH EFNETffzts: quiddity Zppix Tobi_WMDE_SW_NA addshore ema Volker_E thcipriani|afk AaronSchulz RoanKattouw Krenair yurik sir_lestaty MGC greg-g shinken-wm nuria_ jackmcbarn Shanmu [01:14:43] !ops help spam [01:14:46] 11,10▄12,09▄09,06▄05,02▄09,03▄09,08▄11,05▄03,03▄03,05▄13,11▄03,06▄03,08▄02,06▄03,06▄05,07▄02,09▄12,10▄12,09▄02 https://www.youtube.com/watch?v=--jYqzJErTI LRH IS LIVE NOW!! CALL 415-349-5666 #LRH EFNETzwhtbychii: robh MGC marlier icinga-wm ragesoss_ quiddity Shanmugamp7 greg-g addshore Zppix|Away Krenair ema Volker_E RoanKattouw brion tgr|a [01:14:52] 03,07▄11,07▄05,07▄04,04▄05,02▄11,05▄13,12▄11,02▄12,09▄13,03▄06,05▄02 https://www.youtube.com/watch?v=--jYqzJErTI LRH IS LIVE NOW!! CALL 415-349-5666 #LRH EFNETgggsutp: mdholloway brion sbailey phedenskog jackmcbarn ebernhardson TimStarling Reception123 greg-g marlier AaronSchulz thcipriani|afk Urbane RoanKattouw Zppix yurik tgr|away icinga-wm Shanmugamp7 addshore sir_ [01:14:56] 13,03▄06,05▄05,04▄13,06▄08,04▄10,10▄08,02▄12,05▄07,08▄12,07▄10,05▄09,10▄13,07▄13 https://www.youtube.com/watch?v=--jYqzJErTI LRH IS LIVE NOW!! CALL 415-349-5666 #LRH EFNETtxtai: Volker_E thcipriani|afk tgr|away jackmcbarn FlorianSW robh Tobi_WMDE_SW_NA ema Shanmugamp7 shinken-wm MGC sir_lestaty sbailey phedenskog gilles ragesoss_ marlier greg-g Zppix|Away Kren [01:15:02] 03,13▄05,11▄12,03▄02,08▄11,04▄07,02▄09,07▄02,05▄06,04▄08,05▄12,11▄04,05▄09,02▄02,02▄08,04▄13,11▄02,04▄04,05▄07 https://www.youtube.com/watch?v=--jYqzJErTI LRH IS LIVE NOW!! CALL 415-349-5666 #LRH EFNETmrshwzznn: shinken-wm RoanKattouw brion Volker_E nuria_ godog jackmcbarn Zppix|Away phedenskog icinga-wm yurik TimStarling sbailey Zppix Tobi [01:15:07] 08,04▄07,08▄07,06▄02,08▄12,05▄03,03▄12,02▄05,12▄11,10▄06,05▄08,12▄05,12▄13,08▄11 https://www.youtube.com/watch?v=--jYqzJErTI LRH IS LIVE NOW!! CALL 415-349-5666 #LRH EFNETxzllmafg: Krenair brion Zppix jackmcbarn robh AaronSchulz eddiegp Tobi_WMDE_SW_NA Volker_E RoanKattouw addshore mdholloway phedenskog Shanmugamp7 TimStarling ema yurik greg-g gilles ebernhard [01:15:12] 09,06▄13,04▄08,11▄09,10▄10,06▄07,05▄02,13▄11,07▄08,05▄02,03▄08,12▄11,13▄13,09▄11,03▄11,08▄12,03▄04,08▄08,03▄07,13▄04 https://www.youtube.com/watch?v=--jYqzJErTI LRH IS LIVE NOW!! CALL 415-349-5666 #LRH EFNETtsakjnujes: Volker_E robh thcipriani|afk Zppix|Away eddiegp shinken-wm Shanmugamp7 marlier icinga-wm quiddity Tobi_WMDE_SW_NA sbail [01:15:17] 13,02▄04,06▄13,04▄08,04▄09,12▄02,13▄04,10▄05,08▄05,11▄10,10▄02,02▄06,13▄11,05▄03,04▄07,02▄04,09▄02 https://www.youtube.com/watch?v=--jYqzJErTI LRH IS LIVE NOW!! CALL 415-349-5666 #LRH EFNETaemjyu: Zppix|Away greg-g Krenair Urbane sbailey FlorianSW sir_lestaty jackmcbarn phedenskog godog shinken-wm tgr|away robh ragesoss_ ebernhardson brion Volker_E [01:15:22] 03,11▄11,12▄11,08▄10,12▄08,10▄08,02▄12,06▄10,08▄02,13▄03,08▄12 https://www.youtube.com/watch?v=--jYqzJErTI LRH IS LIVE NOW!! CALL 415-349-5666 #LRH EFNETdxmpyugi: quiddity Krenair AaronSchulz Volker_E MGC FlorianSW sir_lestaty phedenskog marlier Shanmugamp7 brion shinken-wm robh Tobi_WMDE_SW_NA godog Zppix|Away Urbane thcipriani|afk nuria_ sbailey addshore gilles yurik ic [01:15:27] 13,11▄07,04▄09,08▄03,13▄05,13▄05,03▄03,06▄07,06▄08,02▄10,03▄03,02▄05,06▄04,12▄02,08▄12,06▄13,13▄11,04▄06,03▄13 https://www.youtube.com/watch?v=--jYqzJErTI LRH IS LIVE NOW!! CALL 415-349-5666 #LRH EFNETraphgk: ema fmla_dr0ptp4kt shinken-wm thcipriani|afk ebernhardson robh godog RoanKattouw AaronSchulz icinga-wm ragesoss_ sbailey Krenair TimS [01:15:32] 05,11▄06,10▄06,09▄08,06▄09,03▄09,10▄05,06▄12,10▄02,05▄05,03▄09,07▄10,07▄08,12▄05,06▄11,03▄11,06▄05,10▄11 https://www.youtube.com/watch?v=--jYqzJErTI LRH IS LIVE NOW!! CALL 415-349-5666 #LRH EFNETzkyxeixsd: jackmcbarn godog icinga-wm AaronSchulz FlorianSW greg-g phedenskog Volker_E mdholloway robh TimStarling addshore marlier Shanmugamp7 Krenair [01:15:36] 09,04▄04,04▄13,02▄12,13▄13,03▄02,10▄02,04▄05,05▄05,06▄08,11▄03,03▄04,07▄07,06▄09,11▄09,12▄09,06▄11,11▄09,05▄04 https://www.youtube.com/watch?v=--jYqzJErTI LRH IS LIVE NOW!! CALL 415-349-5666 #LRH EFNETavzttwi: tgr|away FlorianSW sbailey brion Shanmugamp7 Krenair jackmcbarn godog marlier ragesoss_ fmla_dr0ptp4kt phedenskog TimStarling Zppix [01:50:06] RECOVERY - Puppet errors on deployment-sca01 is OK: OK: Less than 1.00% above the threshold [0.0] [02:08:59] 10Gerrit, 10Release-Engineering-Team: Unable to log in into gerrit: "Cannot assign user name "ashley" to account ; name already in use." - https://phabricator.wikimedia.org/T181945#3843564 (10Paladox) p:05Triage>03High Setting this to high so that it is more noticeable :). [04:48:38] PROBLEM - Free space - all mounts on deployment-fluorine02 is CRITICAL: CRITICAL: deployment-prep.deployment-fluorine02.diskspace._srv.byte_percentfree (<44.44%) [04:49:11] PROBLEM - Mediawiki Error Rate on graphite-labs is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [10.0] [05:14:09] PROBLEM - Mediawiki Error Rate on graphite-labs is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [10.0] [05:40:55] PROBLEM - Puppet errors on saucelabs-01 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [06:20:54] RECOVERY - Puppet errors on saucelabs-01 is OK: OK: Less than 1.00% above the threshold [0.0] [06:35:31] PROBLEM - Puppet errors on deployment-db03 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [07:08:40] RECOVERY - Free space - all mounts on deployment-fluorine02 is OK: OK: All targets OK [07:10:30] RECOVERY - Puppet errors on deployment-db03 is OK: OK: Less than 1.00% above the threshold [0.0] [08:08:55] 10MediaWiki-Codesniffer, 10Patch-For-Review: Upgrade to squizlabs/php_codesniffer 3.2.0 - https://phabricator.wikimedia.org/T183074#3843784 (10Legoktm) I was concerned with the fix for IDE plugins, but he just released 3.2.1 (https://github.com/squizlabs/PHP_CodeSniffer/releases/tag/3.2.1) so that looks good t... [08:09:11] 10MediaWiki-Codesniffer, 10Patch-For-Review: Upgrade to squizlabs/php_codesniffer 3.2.1 - https://phabricator.wikimedia.org/T183074#3843785 (10Legoktm) [08:09:32] (03CR) 10Legoktm: [C: 04-1] "https://github.com/squizlabs/PHP_CodeSniffer/releases/tag/3.2.1 let's go straight to 3.2.1 and I think we can merge this for the next rele" [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/398050 (https://phabricator.wikimedia.org/T183074) (owner: 10Ricordisamoa) [08:33:41] 10Continuous-Integration-Infrastructure (shipyard), 10Release-Engineering-Team (Kanban), 10Operations, 10Ops-Access-Requests, 10Patch-For-Review: Allow contint-admins to interact with docker on CI hosts - https://phabricator.wikimedia.org/T182860#3843804 (10Volans) p:05Triage>03Normal [08:49:34] 10MediaWiki-Codesniffer, 10Patch-For-Review: ClassMatchesFilename.WrongCase should ignore maintenance scripts - https://phabricator.wikimedia.org/T178725#3843818 (10Legoktm) Maintenance scripts should really only have one class in them, so it seems unlikely that the same file will have multiple wrong case issu... [08:49:40] (03CR) 10Legoktm: [C: 04-1] Add sniff for using is_int over is_integer [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/398677 (https://phabricator.wikimedia.org/T173175) (owner: 10Legoktm) [09:30:16] 10Gerrit, 10Release-Engineering-Team: Unable to log in into gerrit: "Cannot assign user name "ashley" to account ; name already in use." - https://phabricator.wikimedia.org/T181945#3843949 (10Aklapper) @Paladox: The Priority field does **not** express noticeability. It expresses urgency. [Nothing... [09:53:42] !log gerrit: deleted wikibase/data-model and wikibase/data-model-services . They are on Github https://github.com/wmde/WikibaseDataModel and https://github.com/wmde/WikibaseDataModelServices [09:53:47] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [09:58:18] !log gerrit: deleted unused user-metrics-2 repo. Been created 4 years and 7 months ago but otherwise unused [09:58:23] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [10:16:26] 10Release-Engineering-Team, 10Scap, 10ORES, 10Operations, and 2 others: Connection timeout from tin to new ores servers - https://phabricator.wikimedia.org/T181661#3844074 (10akosiaris) [10:17:58] 10Scap, 10Operations, 10Patch-For-Review: scap 3.7.4-2 is broken - https://phabricator.wikimedia.org/T183046#3841983 (10akosiaris) [10:18:01] 10Scap, 10Operations, 10Packaging, 10Patch-For-Review: SCAP: Upload debian package version 3.7.4-1 - https://phabricator.wikimedia.org/T182347#3844077 (10akosiaris) [10:27:20] 10Continuous-Integration-Infrastructure, 10Browser-Tests-Infrastructure, 10Packaging, 10Patch-For-Review: Create a Debian package for https://github.com/mozilla/geckodriver for at least Debian Jessie - https://phabricator.wikimedia.org/T137797#3844105 (10zeljkofilipin) [10:27:22] 10Release-Engineering-Team (Kanban), 10Patch-For-Review, 10User-zeljkofilipin: Run WebdriverIO tests using Firefox - https://phabricator.wikimedia.org/T161697#3844104 (10zeljkofilipin) [11:03:59] 10Continuous-Integration-Config, 10Release-Engineering-Team (Kanban), 10Cloud-VPS: Add CI to sink_nova_ldap and sink_nova_fixed_multi - https://phabricator.wikimedia.org/T183129#3844196 (10hashar) [11:04:33] 10Continuous-Integration-Config, 10Release-Engineering-Team (Kanban), 10Cloud-VPS: Add CI to sink_nova_ldap and sink_nova_fixed_multi - https://phabricator.wikimedia.org/T183129#3844208 (10hashar) [11:10:03] (03PS1) 10Hashar: Add sink_nova_ldap and sink_nova_fixed_multi [integration/config] - 10https://gerrit.wikimedia.org/r/398810 (https://phabricator.wikimedia.org/T183129) [11:10:32] (03CR) 10Hashar: [C: 032] Add sink_nova_ldap and sink_nova_fixed_multi [integration/config] - 10https://gerrit.wikimedia.org/r/398810 (https://phabricator.wikimedia.org/T183129) (owner: 10Hashar) [11:12:13] (03Merged) 10jenkins-bot: Add sink_nova_ldap and sink_nova_fixed_multi [integration/config] - 10https://gerrit.wikimedia.org/r/398810 (https://phabricator.wikimedia.org/T183129) (owner: 10Hashar) [11:14:09] 10Continuous-Integration-Config, 10Release-Engineering-Team (Kanban), 10Cloud-VPS, 10Patch-For-Review: Add CI to sink_nova_ldap and sink_nova_fixed_multi - https://phabricator.wikimedia.org/T183129#3844226 (10hashar) p:05Triage>03Normal [11:25:28] (03PS1) 10Hashar: Use UNSTABLE for integration-config-qa [integration/config] - 10https://gerrit.wikimedia.org/r/398816 [11:27:35] PROBLEM - Puppet errors on deployment-trending01 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [11:32:34] (03PS2) 10Hashar: Use UNSTABLE for integration-config-qa [integration/config] - 10https://gerrit.wikimedia.org/r/398816 [11:33:06] (03CR) 10Thiemo Kreuz (WMDE): "I would like to suggest disallowing a bunch of other aliases, too. My suggestion:" [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/398677 (https://phabricator.wikimedia.org/T173175) (owner: 10Legoktm) [11:33:14] (03CR) 10Hashar: [C: 032] Use UNSTABLE for integration-config-qa [integration/config] - 10https://gerrit.wikimedia.org/r/398816 (owner: 10Hashar) [11:34:27] (03Merged) 10jenkins-bot: Use UNSTABLE for integration-config-qa [integration/config] - 10https://gerrit.wikimedia.org/r/398816 (owner: 10Hashar) [11:35:20] (03PS1) 10Hashar: Register some Prometheus exporters [integration/config] - 10https://gerrit.wikimedia.org/r/398817 [11:35:38] (03CR) 10Hashar: [C: 032] Register some Prometheus exporters [integration/config] - 10https://gerrit.wikimedia.org/r/398817 (owner: 10Hashar) [11:36:50] (03Merged) 10jenkins-bot: Register some Prometheus exporters [integration/config] - 10https://gerrit.wikimedia.org/r/398817 (owner: 10Hashar) [11:37:06] 10Scap, 10Operations, 10Patch-For-Review: scap 3.7.4-2 is broken - https://phabricator.wikimedia.org/T183046#3841983 (10akosiaris) So, I 'll add my considerable amounts of cents in this task in order to provide insight into what happened, see where did went south and figure out what we need to do to avoid it... [11:39:46] 10Scap, 10Operations, 10Packaging, 10Patch-For-Review: SCAP: Upload debian package version 3.7.4-3 - https://phabricator.wikimedia.org/T182347#3844304 (10akosiaris) [11:53:57] 10Scap, 10Operations, 10Patch-For-Review: scap 3.7.4-2 is broken - https://phabricator.wikimedia.org/T183046#3841983 (10ArielGlenn) Agree about building 3.7.4-3 for new upload. As far as the bin_dir, have you seen this? https://gerrit.wikimedia.org/r/#/c/398606/ [12:02:56] 10Scap, 10Operations, 10Patch-For-Review: scap 3.7.4-2 is broken - https://phabricator.wikimedia.org/T183046#3844359 (10akosiaris) The 2 first actionables are in D919. I 've also merged the bin_dir configuration change above, but that's just a safeguard, as per the comment in the change "Ideally, we want t... [12:13:02] 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Watching / External), 10ChangeProp, and 3 others: Setup change-propagation service CI - https://phabricator.wikimedia.org/T152684#3844379 (10mobrovac) [12:13:15] 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Someday), 10Reading-Infrastructure-Team-Backlog, and 3 others: Setup trending service CI - https://phabricator.wikimedia.org/T151469#3844381 (10mobrovac) [12:13:18] 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Watching / External), 10ChangeProp, and 3 others: Setup change-propagation service CI - https://phabricator.wikimedia.org/T152684#2857076 (10mobrovac) [12:14:15] 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Someday), 10Reading-Infrastructure-Team-Backlog, and 3 others: Setup trending service CI - https://phabricator.wikimedia.org/T151469#2818282 (10mobrovac) [12:49:28] 10Release-Engineering-Team (Kanban), 10User-zeljkofilipin: Prepare for All Hands workshop - https://phabricator.wikimedia.org/T183134#3844430 (10zeljkofilipin) [12:49:43] 10Release-Engineering-Team (Kanban), 10User-zeljkofilipin: Prepare for All Hands workshop - https://phabricator.wikimedia.org/T183134#3844442 (10zeljkofilipin) p:05Triage>03Low [12:55:34] 10MediaWiki-Codesniffer, 10Patch-For-Review: ClassMatchesFilename.WrongCase should ignore maintenance scripts - https://phabricator.wikimedia.org/T178725#3844454 (10Huji) Fair. Let's leave it as is. [12:56:12] (03PS1) 10Hashar: Bump wmfreleng/npm-browser-test for Chromium 63 [integration/config] - 10https://gerrit.wikimedia.org/r/398828 [12:57:29] (03CR) 10Hashar: [C: 032] Bump wmfreleng/npm-browser-test for Chromium 63 [integration/config] - 10https://gerrit.wikimedia.org/r/398828 (owner: 10Hashar) [12:58:51] (03Merged) 10jenkins-bot: Bump wmfreleng/npm-browser-test for Chromium 63 [integration/config] - 10https://gerrit.wikimedia.org/r/398828 (owner: 10Hashar) [13:00:16] 10Continuous-Integration-Config, 10MediaWiki-extensions-WikibaseClient, 10Wikidata: Phan did not catch missing use statement in Wikibase patch - https://phabricator.wikimedia.org/T183083#3844458 (10Addshore) Tests are not tested by phan in the wikibase config https://github.com/wikimedia/mediawiki-extension... [13:11:26] (03PS2) 10Hashar: docker: convert zuul-cloner to docker-pkg [integration/config] - 10https://gerrit.wikimedia.org/r/398263 [13:12:01] (03CR) 10Hashar: [C: 032] docker: convert zuul-cloner to docker-pkg [integration/config] - 10https://gerrit.wikimedia.org/r/398263 (owner: 10Hashar) [13:13:07] (03Merged) 10jenkins-bot: docker: convert zuul-cloner to docker-pkg [integration/config] - 10https://gerrit.wikimedia.org/r/398263 (owner: 10Hashar) [13:13:22] 10Release-Engineering-Team (Kanban), 10Discovery, 10Discovery-Search (Current work), 10MW-1.31-release-notes (WMF-deploy-2017-10-24 (1.31.0-wmf.5)), 10Patch-For-Review: [Epic] Port Selenium tests from Ruby to Node.js for the Search Platform - https://phabricator.wikimedia.org/T174103#3844495 (10zeljkofil... [13:47:26] 10Scap, 10Operations, 10Patch-For-Review: scap 3.7.4-2 is broken - https://phabricator.wikimedia.org/T183046#3844602 (10ArielGlenn) The control file seems to be wrong. It should be this https://phabricator.wikimedia.org/source/scap/browse/master/debian/control because of a move of python-semver from depen... [14:08:09] (03PS1) 10Hashar: Switch ci-src-setup-simple to our docker registry [integration/config] - 10https://gerrit.wikimedia.org/r/398842 [14:10:46] (03PS2) 10Hashar: docker: ci-src-setup-simple based on our docker registry [integration/config] - 10https://gerrit.wikimedia.org/r/398842 [14:26:22] 10Release-Engineering-Team (Kanban), 10Discovery, 10Discovery-Search (Current work), 10Patch-For-Review, 10User-zeljkofilipin: Run selenium-EXTENSION-jessie Jenkins job for CirrusSearch - https://phabricator.wikimedia.org/T175179#3844751 (10zeljkofilipin) @dcausse I am still stuck. I do not know how to r... [14:27:58] 10Release-Engineering-Team (Kanban), 10Discovery, 10Discovery-Search (Current work), 10Patch-For-Review, 10User-zeljkofilipin: Run selenium-EXTENSION-jessie Jenkins job for CirrusSearch - https://phabricator.wikimedia.org/T175179#3844756 (10zeljkofilipin) ``` $ CIRRUS_BETA_SMOKE_TEST=true ./node_modules/... [14:35:00] (03PS1) 10Hashar: docker: ci-stretch missed DEBIAN_FRONTEND=noninteractive [integration/config] - 10https://gerrit.wikimedia.org/r/398846 [14:35:34] (03CR) 10Hashar: [C: 032] docker: ci-stretch missed DEBIAN_FRONTEND=noninteractive [integration/config] - 10https://gerrit.wikimedia.org/r/398846 (owner: 10Hashar) [14:37:08] (03Merged) 10jenkins-bot: docker: ci-stretch missed DEBIAN_FRONTEND=noninteractive [integration/config] - 10https://gerrit.wikimedia.org/r/398846 (owner: 10Hashar) [14:40:26] 10MediaWiki-Releasing, 10MediaWiki-Vendor: mediawiki/vendor is bloated for tarball releases - https://phabricator.wikimedia.org/T172927#3513508 (10Seb35) I was about to ask if there is a documentation/policy about what is included in mediawiki/vendor compared to mediawiki/core, and then I found this task. If m... [14:46:48] 10Scap, 10Operations, 10Patch-For-Review: scap 3.7.4-2 is broken - https://phabricator.wikimedia.org/T183046#3844824 (10akosiaris) >>! In T183046#3844602, @ArielGlenn wrote: > The control file seems to be wrong. It should be this https://phabricator.wikimedia.org/source/scap/browse/master/debian/control be... [14:49:23] no_justification hi, im wondering do we want to include the gelf dep in gerrit (our fork). Because upstream doint seem to be merging it anytime soon as they want to fix it so that you you can use the lib folder. [14:49:31] Also this will fix it so it includes a trace back [14:49:47] which currently it seems log4j socket appender dosen't seem to be doing. [14:51:30] 10Scap, 10Operations, 10Patch-For-Review: scap 3.7.4-2 is broken - https://phabricator.wikimedia.org/T183046#3841983 (10thcipriani) the documentation on the prep-side for packages is at https://wikitech.wikimedia.org/wiki/How_to_deploy_code/Scap which should probably get combined with https://wikitech.wikime... [14:54:47] (03PS3) 10Ricordisamoa: Update PHP_CodeSniffer to 3.2.1 [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/398050 [14:56:24] (03PS4) 10Ricordisamoa: Update PHP_CodeSniffer to 3.2.1 [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/398050 [14:58:22] (03PS1) 10Hashar: fabfile: env.sudo_user => settings(sudo_user=) [integration/config] - 10https://gerrit.wikimedia.org/r/398852 [14:59:14] (03CR) 10Hashar: "That is an issue I had with deploy_docker which changed to zuul when updating integration/config and would thus later try to run docker-pk" [integration/config] - 10https://gerrit.wikimedia.org/r/398852 (owner: 10Hashar) [15:03:02] no_justification it will be really easy to maintain the dep :). Just need to cherry pick my patch over to the fork. [15:08:56] (03PS5) 10Ricordisamoa: Update PHP_CodeSniffer to 3.2.1 [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/398050 (https://phabricator.wikimedia.org/T183074) [15:13:54] 10Release-Engineering-Team (Kanban), 10Discovery, 10Discovery-Search (Current work), 10Patch-For-Review, 10User-zeljkofilipin: Run selenium-EXTENSION-jessie Jenkins job for CirrusSearch - https://phabricator.wikimedia.org/T175179#3845045 (10EBernhardson) chomedriver has to be started to listen on :4444.... [15:15:01] PROBLEM - Puppet errors on deployment-redis06 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [15:18:40] PROBLEM - English Wikipedia Main page on beta-cluster is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:23:30] RECOVERY - English Wikipedia Main page on beta-cluster is OK: HTTP OK: HTTP/1.1 200 OK - 47006 bytes in 0.969 second response time [15:26:52] (03Abandoned) 10Hashar: Jenkins job validation (DO NOT SUBMIT) [integration/uprightdiff] - 10https://gerrit.wikimedia.org/r/391210 (owner: 10Hashar) [15:27:06] (03Abandoned) 10Hashar: Build docker images on post merge [integration/config] - 10https://gerrit.wikimedia.org/r/385213 (https://phabricator.wikimedia.org/T178594) (owner: 10Hashar) [15:27:37] (03Abandoned) 10Hashar: docker ignore 'cache' and 'log' files [integration/config] - 10https://gerrit.wikimedia.org/r/381289 (owner: 10Hashar) [15:27:45] PROBLEM - Puppet errors on deployment-redis01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [15:33:42] PROBLEM - Puppet errors on deployment-redis02 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [15:39:12] (03CR) 10Hashar: [C: 032] docker: squash apt-get install in a single RUN [integration/quibble] - 10https://gerrit.wikimedia.org/r/383152 (owner: 10Hashar) [15:39:14] (03CR) 10Hashar: [C: 032] Running global-setup.sh is useless [integration/quibble] - 10https://gerrit.wikimedia.org/r/383159 (owner: 10Hashar) [15:39:35] (03Merged) 10jenkins-bot: docker: squash apt-get install in a single RUN [integration/quibble] - 10https://gerrit.wikimedia.org/r/383152 (owner: 10Hashar) [15:39:38] (03Merged) 10jenkins-bot: Running global-setup.sh is useless [integration/quibble] - 10https://gerrit.wikimedia.org/r/383159 (owner: 10Hashar) [15:40:10] RECOVERY - Mediawiki Error Rate on graphite-labs is OK: OK: Less than 1.00% above the threshold [1.0] [15:41:47] (03PS2) 10Hashar: Add a note about mw debug file name [integration/jenkins] - 10https://gerrit.wikimedia.org/r/369663 (https://phabricator.wikimedia.org/T50002) [15:41:58] (03CR) 10Hashar: [C: 032] Add a note about mw debug file name [integration/jenkins] - 10https://gerrit.wikimedia.org/r/369663 (https://phabricator.wikimedia.org/T50002) (owner: 10Hashar) [15:43:04] (03Merged) 10jenkins-bot: Add a note about mw debug file name [integration/jenkins] - 10https://gerrit.wikimedia.org/r/369663 (https://phabricator.wikimedia.org/T50002) (owner: 10Hashar) [15:46:50] 10Continuous-Integration-Config, 10Release-Engineering-Team (Kanban), 10Operations, 10Patch-For-Review, 10Puppet: Get rid of "import realm.pp" in manifests/site.pp - https://phabricator.wikimedia.org/T154915#3845179 (10hashar) a:03hashar [15:47:23] 10Continuous-Integration-Config, 10Release-Engineering-Team (Kanban), 10Operations, 10Patch-For-Review, 10Puppet: Get rid of "import realm.pp" in manifests/site.pp - https://phabricator.wikimedia.org/T154915#3845181 (10hashar) [15:47:57] 10Continuous-Integration-Config, 10Release-Engineering-Team (Kanban), 10Operations, 10Patch-For-Review, 10Puppet: Get rid of "import realm.pp" in manifests/site.pp - https://phabricator.wikimedia.org/T154915#2928315 (10hashar) [15:51:14] 10Release-Engineering-Team (Watching / External), 10Operations, 10Patch-For-Review, 10Scoring-platform-team (Current), 10Wikimedia-Incident: Cache ORES virtualenv within versioned source - https://phabricator.wikimedia.org/T181071#3845197 (10awight) @akosiaris Just a nudge, I'm waiting for your feedback... [15:58:07] (03CR) 10Hashar: [C: 04-1] Use --jobs in git submodule update jobs [integration/config] - 10https://gerrit.wikimedia.org/r/391712 (owner: 10Chad) [15:58:31] (03CR) 10Hashar: [C: 04-2] "Should use docker-pkg instead" [integration/config] - 10https://gerrit.wikimedia.org/r/397634 (owner: 10Hashar) [16:33:15] 10Release-Engineering-Team (Kanban), 10Discovery, 10Discovery-Search (Current work), 10Patch-For-Review, 10User-zeljkofilipin: Run selenium-EXTENSION-jessie Jenkins job for CirrusSearch - https://phabricator.wikimedia.org/T175179#3845337 (10zeljkofilipin) Argh. Of course. My mistake. Works fine now. ```... [16:40:38] (03CR) 10Krinkle: "minor fixme/clarification question" (031 comment) [integration/jenkins] - 10https://gerrit.wikimedia.org/r/369663 (https://phabricator.wikimedia.org/T50002) (owner: 10Hashar) [16:46:54] 10Beta-Cluster-Infrastructure: ApiFeatureUsage data is not being populated in the Beta Cluster - https://phabricator.wikimedia.org/T183156#3845398 (10Anomie) [17:08:10] 10Scap, 10Operations, 10Packaging, 10Patch-For-Review: SCAP: Upload debian package version 3.7.4-3 - https://phabricator.wikimedia.org/T182347#3845445 (10akosiaris) 05Open>03Resolved Done, re-resolving finally [17:08:12] 10Release-Engineering-Team, 10Scap, 10ORES, 10Operations, and 2 others: Connection timeout from tin to new ores servers - https://phabricator.wikimedia.org/T181661#3845447 (10akosiaris) [17:08:51] 10Release-Engineering-Team (Kanban), 10Discovery, 10Discovery-Search (Current work), 10Patch-For-Review, 10User-zeljkofilipin: Run selenium-EXTENSION-jessie Jenkins job for CirrusSearch - https://phabricator.wikimedia.org/T175179#3845448 (10zeljkofilipin) The job is also green! 💚 [[ https://integration.w... [17:25:56] 10Beta-Cluster-Infrastructure: various .beta.wmflabs.org domains use an invalid ssl certificate - https://phabricator.wikimedia.org/T182927#3845486 (10Dzahn) Yea, waiting for Letsencrypt wildcard certs to come seems the right way here. It should hopefully be soon. [17:36:10] 10Gerrit, 10Release-Engineering-Team (Kanban), 10Patch-For-Review, 10Zuul: Update zuul to upstream master - https://phabricator.wikimedia.org/T158243#3845507 (10greg) [17:36:24] !log paused beta-code-update-eqiad for a while while I test something [17:36:25] 10Gerrit, 10Release-Engineering-Team (Kanban), 10Patch-For-Review, 10Zuul: Update zuul to upstream master - https://phabricator.wikimedia.org/T158243#3030929 (10greg) a:03hashar [17:36:30] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [17:36:37] 10Release-Engineering-Team (Kanban), 10User-zeljkofilipin: Sample code in Node.js for repositories that still have Selenium+Ruby tests - https://phabricator.wikimedia.org/T183160#3845511 (10zeljkofilipin) [17:36:54] 10Release-Engineering-Team (Kanban), 10User-zeljkofilipin: Q3 Selenium framework improvements - https://phabricator.wikimedia.org/T182421#3845521 (10zeljkofilipin) [17:36:57] 10Release-Engineering-Team (Kanban), 10User-zeljkofilipin: Sample code in Node.js for repositories that still have Selenium+Ruby tests - https://phabricator.wikimedia.org/T183160#3845520 (10zeljkofilipin) [17:37:07] 10Release-Engineering-Team (Kanban), 10User-zeljkofilipin: Sample code in Node.js for repositories that still have Selenium+Ruby tests - https://phabricator.wikimedia.org/T183160#3845511 (10zeljkofilipin) p:05Triage>03Normal [17:37:58] 10Gerrit, 10Release-Engineering-Team (Kanban), 10Patch-For-Review, 10Zuul: Update zuul to upstream master - https://phabricator.wikimedia.org/T158243#3845537 (10hashar) That is blocking the upgrade of Gerrit from 2.13 to 2.14 which is overdue now. Spoke with @demon about it during the weekly #together ch... [17:38:11] 10Gerrit, 10Release-Engineering-Team (Kanban), 10Patch-For-Review, 10Zuul: Update zuul to upstream master - https://phabricator.wikimedia.org/T158243#3845538 (10hashar) p:05Normal>03High [17:39:22] 10Release-Engineering-Team (Kanban), 10User-zeljkofilipin: Patches in Gerrit deleting Selenium+Ruby tests for repositories that still have them - https://phabricator.wikimedia.org/T183162#3845542 (10zeljkofilipin) [17:39:35] 10Release-Engineering-Team (Kanban), 10User-zeljkofilipin: Q3 Selenium framework improvements - https://phabricator.wikimedia.org/T182421#3845552 (10zeljkofilipin) [17:39:37] 10Release-Engineering-Team (Kanban), 10User-zeljkofilipin: Patches in Gerrit deleting Selenium+Ruby tests for repositories that still have them - https://phabricator.wikimedia.org/T183162#3845551 (10zeljkofilipin) [17:39:48] 10Release-Engineering-Team (Kanban), 10User-zeljkofilipin: Patches in Gerrit deleting Selenium+Ruby tests for repositories that still have them - https://phabricator.wikimedia.org/T183162#3845542 (10zeljkofilipin) p:05Triage>03Normal [17:40:37] 10Release-Engineering-Team (Kanban), 10releng-201718-q1, 10MediaWiki-General-or-Unknown, 10Epic, and 5 others: Port Selenium tests from Ruby to Node.js - https://phabricator.wikimedia.org/T139740#3845556 (10zeljkofilipin) [17:40:40] 10Release-Engineering-Team (Someday), 10Differential, 10Wikibugs, 10Gerrit-Migration: Broadcast Differential activity to IRC - https://phabricator.wikimedia.org/T116330#3845559 (10greg) a:05mmodell>03None [17:40:48] 10Release-Engineering-Team (Kanban), 10Release Pipeline, 10Patch-For-Review: Define new Jenkins pipeline for container build phase - https://phabricator.wikimedia.org/T175297#3845561 (10dduvall) 05Open>03Resolved [17:41:24] 10Scap: cache_git_info (from e.g. scap sync-file) is way way too verbose - https://phabricator.wikimedia.org/T182643#3829796 (10Addshore) It looks like this is fixed on beta at least now... ``` jenkins-deploy@deployment-tin:/srv/mediawiki-staging/php-master$ scap sync-file php-master/includes/Storage/RevisionS... [17:41:51] (03PS2) 10Hashar: docker: php5.5 images [integration/config] - 10https://gerrit.wikimedia.org/r/397634 [17:42:04] 10Release-Engineering-Team, 10Epic: FY2017/18 Program 3 Outcome 1: The amount of orphaned code that is running Wikimedia “production” services is reduced. - https://phabricator.wikimedia.org/T174088#3845569 (10greg) [17:43:08] 10Release-Engineering-Team (Watching / External), 10MediaWiki-Core-Tests, 10Patch-For-Review, 10User-zeljkofilipin: WebdriverIO should run Chrome headlessly - https://phabricator.wikimedia.org/T167507#3845576 (10zeljkofilipin) [17:43:42] 10Release-Engineering-Team (Kanban), 10Patch-For-Review, 10User-zeljkofilipin: Run WebdriverIO tests using Firefox - https://phabricator.wikimedia.org/T161697#3845577 (10zeljkofilipin) [17:44:18] (03PS3) 10Hashar: docker: php5.5 images [integration/config] - 10https://gerrit.wikimedia.org/r/397634 [17:44:57] (03CR) 10Hashar: [C: 04-2] "Converted to docker-pkg. Would like to migrate a few jobs while at it." [integration/config] - 10https://gerrit.wikimedia.org/r/397634 (owner: 10Hashar) [17:45:26] 10Release-Engineering-Team (Kanban), 10Scap (Tech Debt Sprint FY201718-Q2): Audit dblists to see if they're all needed - https://phabricator.wikimedia.org/T179972#3845587 (10greg) p:05High>03Low [17:45:38] 10Release-Engineering-Team (Kanban), 10User-zeljkofilipin: Docker container with geckodriver - https://phabricator.wikimedia.org/T183163#3845588 (10zeljkofilipin) [17:45:57] 10Release-Engineering-Team (Kanban), 10User-zeljkofilipin: Docker container with geckodriver - https://phabricator.wikimedia.org/T183163#3845598 (10zeljkofilipin) [17:46:01] 10Release-Engineering-Team (Kanban), 10Patch-For-Review, 10User-zeljkofilipin: Run WebdriverIO tests using Firefox - https://phabricator.wikimedia.org/T161697#3845599 (10zeljkofilipin) [17:48:29] 10Continuous-Integration-Infrastructure (shipyard), 10Release-Engineering-Team (Watching / External), 10Operations, 10Release Pipeline: Switch CI Docker Storage Driver to devicemapper - https://phabricator.wikimedia.org/T178663#3845604 (10greg) [17:50:44] 10Release-Engineering-Team (Kanban), 10User-zeljkofilipin: Docker container with geckodriver - https://phabricator.wikimedia.org/T183163#3845608 (10zeljkofilipin) p:05Triage>03Normal [17:51:32] 10Release-Engineering-Team (Kanban), 10Release Pipeline (Blubber): Support for Blubber defaults and/or policies - https://phabricator.wikimedia.org/T174631#3845612 (10greg) 05Open>03declined talked it through and base images seems like plenty. If there is a need for something like this in the future it wo... [17:52:43] 10Release-Engineering-Team (Kanban), 10Patch-For-Review, 10Release Pipeline (Blubber): Establish blubber config in mathoid - https://phabricator.wikimedia.org/T173127#3845625 (10thcipriani) 05Open>03Resolved we've got a blubber.yaml in mathiod now [17:53:02] 10Release-Engineering-Team (Kanban), 10Mathoid, 10Release Pipeline, 10Patch-For-Review: Add experimental blubber test build/run to mathoid jenkins test pipeline - https://phabricator.wikimedia.org/T177954#3845628 (10dduvall) 05Open>03Resolved [17:53:52] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban), 10Jenkins: Move the beta cluster jobs to a dedicated/standalone Jenkins instance - https://phabricator.wikimedia.org/T183164#3845630 (10greg) p:05Triage>03Normal [17:54:13] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Watching / External), 10Jenkins, 10Upstream: Jenkins Gearman plugin has deadlock on executor threads (was: Beta Cluster stopped receiving code updates (beta-update-databases-eqiad hung) - https://phabricator.wikimedia.org/T72597#3845643 (1... [17:56:56] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team (Kanban): Various puppet issues in deployment-prep - https://phabricator.wikimedia.org/T180935#3845646 (10greg) a:03hashar @hashar Let's break out the remaining issues from this task into subtasks of {T132259} and close this one (they're kind of redun... [17:57:53] 10Release-Engineering-Team (Kanban), 10Scap (Tech Debt Sprint FY201718-Q2), 10scap2: scap3 should repack / pack-refs git repos under /srv/deployment - https://phabricator.wikimedia.org/T112509#3845653 (10greg) @mmodell will take a look at this again :) [17:59:18] 10Gerrit, 10Release-Engineering-Team (Kanban): Unable to log in into gerrit: "Cannot assign user name "ashley" to account ; name already in use." - https://phabricator.wikimedia.org/T181945#3845655 (10greg) [18:00:08] 10Continuous-Integration-Infrastructure (shipyard), 10Release-Engineering-Team (Kanban), 10Operations, 10Ops-Access-Requests, 10Patch-For-Review: Allow contint-admins to interact with docker on CI hosts - https://phabricator.wikimedia.org/T182860#3845657 (10RobH) Please note this was provisionally approv... [18:00:38] 10Release-Engineering-Team, 10Maps-Sprint, 10Repository-Admins, 10Maps (Tilerator): Setup diffusion and github sync for kartotherian and tilerator package repositories - https://phabricator.wikimedia.org/T182848#3845659 (10greg) [18:02:32] 10Release-Engineering-Team (Someday), 10GitHub-Mirrors: Audit @wikimedia GitHub org access (2017) - https://phabricator.wikimedia.org/T179462#3845661 (10greg) p:05Triage>03Low [18:05:59] 10Release-Engineering-Team (Kanban), 10releng-201718-q1, 10Release Pipeline: Provide gating feedback from mathoid image builds - https://phabricator.wikimedia.org/T170483#3845666 (10thcipriani) 05Open>03declined Task is a bit ambiguous and will be continually addressed as we move forward with the pipelin... [18:08:16] 10Release-Engineering-Team (Kanban), 10Release Pipeline: Verify functionality of the 'production' image in the context of an isolated k8s deployment - https://phabricator.wikimedia.org/T183165#3845669 (10thcipriani) [18:09:16] 10Release-Engineering-Team (Kanban), 10Release Pipeline: Verify functionality of the 'production' image in the context of an isolated k8s deployment - https://phabricator.wikimedia.org/T183165#3845681 (10thcipriani) [18:10:10] (03CR) 10Legoktm: [C: 032] "Thanks!" [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/398050 (https://phabricator.wikimedia.org/T183074) (owner: 10Ricordisamoa) [18:10:33] (03PS2) 10Legoktm: Use upstream Generic.PHP.DiscourageGoto [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/398671 [18:10:52] (03Merged) 10jenkins-bot: Update PHP_CodeSniffer to 3.2.1 [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/398050 (https://phabricator.wikimedia.org/T183074) (owner: 10Ricordisamoa) [18:10:54] (03CR) 10Legoktm: [C: 032] "Per Thiemo" [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/398671 (owner: 10Legoktm) [18:11:41] (03Merged) 10jenkins-bot: Use upstream Generic.PHP.DiscourageGoto [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/398671 (owner: 10Legoktm) [18:12:41] 10MediaWiki-Codesniffer, 10Patch-For-Review: Upgrade to squizlabs/php_codesniffer 3.2.1 - https://phabricator.wikimedia.org/T183074#3845699 (10Legoktm) 05Open>03Resolved [18:15:06] 10Release-Engineering-Team (Watching / External), 10Operations, 10ops-eqdfw: setup/install/deploy deploy1001 as deployment server - https://phabricator.wikimedia.org/T175288#3588945 (10Dzahn) I would take this ticket from the "OS installation" step forward but i am unsure if the hardware troubleshooting part... [18:23:29] 10Release-Engineering-Team (Next), 10GitHub-Mirrors: Audit @wikimedia GitHub org access (2017) - https://phabricator.wikimedia.org/T179462#3845712 (10greg) p:05Low>03Normal [18:24:28] 10Release-Engineering-Team (Kanban), 10Release Pipeline: Verify functionality of the 'production' image in the context of an isolated k8s deployment - https://phabricator.wikimedia.org/T183165#3845715 (10thcipriani) [18:24:57] 10Release-Engineering-Team (Watching / External), 10Operations, 10ops-eqdfw: setup/install/deploy deploy1001 as deployment server - https://phabricator.wikimedia.org/T175288#3588945 (10MoritzMuehlenhoff) We have stretch builds of HHVM and this should work (with some minor changes maybe). We should really pro... [18:39:22] PROBLEM - Puppet errors on deployment-mx is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [18:52:41] Project beta-code-update-eqiad build #185786: 04FAILURE in 48 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/185786/ [18:53:37] Project beta-code-update-eqiad build #185787: 04STILL FAILING in 36 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/185787/ [18:54:13] 10MediaWiki-Codesniffer, 10Patch-For-Review: Handle multiple array types from phpdoc in UnusedUseStatementSniff - https://phabricator.wikimedia.org/T178195#3845798 (10Legoktm) 05Open>03Resolved [19:03:38] Project beta-code-update-eqiad build #185788: 04STILL FAILING in 37 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/185788/ [19:05:16] 19:03:00 INFO:mwcore:cwd: /srv/mediawiki-staging/php-master [19:05:16] 19:03:00 INFO:mwcore:running: git pull [19:05:16] 19:03:01 You are not currently on a branch. [19:05:17] eh [19:09:50] whereami [19:13:41] Project beta-code-update-eqiad build #185789: 04STILL FAILING in 40 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/185789/ [19:16:07] twentyafterfour: ^ [19:20:14] Project beta-update-databases-eqiad build #22122: 04FAILURE in 14 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/22122/ [19:23:39] Project beta-code-update-eqiad build #185790: 04STILL FAILING in 38 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/185790/ [19:32:15] (03PS4) 10Hashar: docker: php5.5 images [integration/config] - 10https://gerrit.wikimedia.org/r/397634 [19:33:38] Project beta-code-update-eqiad build #185791: 04STILL FAILING in 38 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/185791/ [19:34:57] (03PS5) 10Hashar: docker: php5.5 images [integration/config] - 10https://gerrit.wikimedia.org/r/397634 [19:34:59] (03PS1) 10Hashar: Archive sink_nova_ldap and sink_nova_fixed_multi [integration/config] - 10https://gerrit.wikimedia.org/r/398901 (https://phabricator.wikimedia.org/T183129) [19:35:41] (03PS2) 10Hashar: Archive sink_nova_ldap and sink_nova_fixed_multi [integration/config] - 10https://gerrit.wikimedia.org/r/398901 (https://phabricator.wikimedia.org/T183129) [19:35:43] (03CR) 10Hashar: [C: 032] Archive sink_nova_ldap and sink_nova_fixed_multi [integration/config] - 10https://gerrit.wikimedia.org/r/398901 (https://phabricator.wikimedia.org/T183129) (owner: 10Hashar) [19:35:46] 10Continuous-Integration-Config, 10Release-Engineering-Team (Kanban), 10Cloud-VPS, 10Patch-For-Review: Add CI to sink_nova_ldap and sink_nova_fixed_multi - https://phabricator.wikimedia.org/T183129#3845897 (10hashar) 05Open>03Resolved [19:36:40] (03Merged) 10jenkins-bot: Archive sink_nova_ldap and sink_nova_fixed_multi [integration/config] - 10https://gerrit.wikimedia.org/r/398901 (https://phabricator.wikimedia.org/T183129) (owner: 10Hashar) [19:43:04] !log checkout master on deployment-tin:/srv/mediawiki-staging/php-master to fix beta-code-update-eqiad [19:43:09] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [19:43:38] Yippee, build fixed! [19:43:38] Project beta-code-update-eqiad build #185792: 09FIXED in 38 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/185792/ [19:49:37] PROBLEM - Mediawiki Error Rate on graphite-labs is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [10.0] [20:01:13] (03PS6) 10Hashar: docker: php5.5 images [integration/config] - 10https://gerrit.wikimedia.org/r/397634 [20:07:47] 10MediaWiki-Releasing, 10MediaWiki-Vendor: 1.27 tarball: Unnecessary library "ruflin/elastica 2.3.1" requirement - https://phabricator.wikimedia.org/T156637#2982017 (10Seb35) This task is similar to T172927, given that ruflin/elastica is an example of the few librairies that are not necessary for all users. [20:21:40] Yippee, build fixed! [20:21:40] Project beta-update-databases-eqiad build #22123: 09FIXED in 1 min 39 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/22123/ [20:29:28] 10Release-Engineering-Team (Next), 10GitHub-Mirrors: Audit @wikimedia GitHub org access (2017) - https://phabricator.wikimedia.org/T179462#3846070 (10Krinkle) >>! In T179462#3743013, @Jdforrester-WMF wrote: >>>! In T179462#3742531, @greg wrote: >> @Jdforrester-WMF as one of the repos is grunt-tyops > I'd just... [20:34:23] Project selenium-Wikibase-chrome » chrome,beta,Linux,DebianJessie && contintLabsSlave build #46: 04FAILURE in 47 min: https://integration.wikimedia.org/ci/job/selenium-Wikibase-chrome/BROWSER=chrome,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=DebianJessie%20&&%20contintLabsSlave/46/ [20:34:29] PROBLEM - Puppet errors on deployment-eventlogging04 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [20:40:26] 10MediaWiki-Codesniffer: Upgrade to squizlabs/php_codesniffer 3.2.1 - https://phabricator.wikimedia.org/T183074#3846119 (10Ricordisamoa) [20:54:39] RECOVERY - Mediawiki Error Rate on graphite-labs is OK: OK: Less than 1.00% above the threshold [1.0] [21:10:03] (03PS7) 10Hashar: docker: php5.5 images [integration/config] - 10https://gerrit.wikimedia.org/r/397634 [21:14:38] (03PS1) 10Hashar: Migrate composer php55 jobs to Docker [integration/config] - 10https://gerrit.wikimedia.org/r/398920 [21:14:40] (03PS1) 10Hashar: Migrate composer php70 jobs to Docker [integration/config] - 10https://gerrit.wikimedia.org/r/398921 [21:20:03] Project beta-update-databases-eqiad build #22124: 04FAILURE in 2.4 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/22124/ [21:28:44] PROBLEM - English Wikipedia Main page on beta-cluster is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 23770 bytes in 0.749 second response time [21:28:50] PROBLEM - App Server Main HTTP Response on deployment-mediawiki06 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 23217 bytes in 0.670 second response time [21:30:37] PROBLEM - App Server Main HTTP Response on deployment-mediawiki04 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 23217 bytes in 0.745 second response time [21:31:15] PROBLEM - English Wikipedia Mobile Main page on beta-cluster is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 19229 bytes in 0.873 second response time [21:31:33] PROBLEM - App Server Main HTTP Response on deployment-mediawiki05 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 23217 bytes in 0.579 second response time [22:00:23] 10Gerrit, 10Operations: Can't fetch from gerrit after updating ssh keys - https://phabricator.wikimedia.org/T183193#3846450 (10Dzahn) [22:03:27] (03PS1) 10Ricordisamoa: Replace PEAR with Packagist in README.md link [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/398945 [22:04:30] 10Gerrit, 10Operations: Can't fetch from gerrit after updating ssh keys - https://phabricator.wikimedia.org/T183193#3846502 (10Dzahn) I noticed the line ``` debug1: Executing proxy command: exec ssh -a -W gerrit.wikimedia.org:29418 bast1001.wikimedia.org ``` But it should directly connect to gerrit on that... [22:07:09] greg-g: Any specific date/time for the window or just "ping Chad and do it whenever he has time"? :) [22:08:03] eddiegp: that's how he works best, I've found :P [22:08:34] greg-g: Fine with me ;) [22:08:39] And thanks for the response! :) [22:12:25] (03CR) 10Ricordisamoa: "There are some more PEAR URLs in the repository you may want to check" [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/398945 (owner: 10Ricordisamoa) [22:20:02] Project beta-update-databases-eqiad build #22125: 04STILL FAILING in 2.3 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/22125/ [22:20:15] PROBLEM - Puppet errors on deployment-sentry01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [22:25:52] could somebody explain to me why phan fails on code that hasn't even been touched in the patch? https://integration.wikimedia.org/ci/job/mwext-php70-phan-docker/1744/console [22:26:05] isn't it supposed to be checking only updated code? [22:33:43] (03PS1) 10Hashar: Migrate composer hhvm jobs to Docker [integration/config] - 10https://gerrit.wikimedia.org/r/399107 [22:35:42] PROBLEM - Puppet errors on deployment-etcd-01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [22:39:12] (03PS8) 10Hashar: docker: php5.5 images [integration/config] - 10https://gerrit.wikimedia.org/r/397634 [22:39:14] (03PS2) 10Hashar: Migrate composer php55 jobs to Docker [integration/config] - 10https://gerrit.wikimedia.org/r/398920 [22:39:16] (03PS2) 10Hashar: Migrate composer php70 jobs to Docker [integration/config] - 10https://gerrit.wikimedia.org/r/398921 [22:39:18] (03PS2) 10Hashar: Migrate composer hhvm jobs to Docker [integration/config] - 10https://gerrit.wikimedia.org/r/399107 [22:40:26] (03PS1) 10Hashar: gitignore docker-pkg-build.log [integration/config] - 10https://gerrit.wikimedia.org/r/399108 [22:40:39] SMalyshev: I think it's bawolff who would know best [22:41:50] SMalyshev: phan checks the entire code base [22:42:24] SMalyshev: you can add 'PhanDeprecatedFunction' to the ignore list for now, or use @suppress I think [22:49:39] 10Gerrit, 10Operations: Can't fetch from gerrit after updating ssh keys - https://phabricator.wikimedia.org/T183193#3846731 (10kaldari) I already added the key via the Gerrit web UI and confirmed that it's there. I just now updated my ssh config to the following: ``` Host bast1001.wikimedia.org ProxyComma... [23:06:26] is grafana in beta cluster alive? All metrics for ores are zero: https://grafana-labs.wikimedia.org/dashboard/db/ores-beta-cluster?orgId=1&from=now-7d&to=now https://grafana-labs.wikimedia.org/dashboard/db/ores-extension?orgId=1 [23:20:03] Project beta-update-databases-eqiad build #22126: 04STILL FAILING in 3 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/22126/ [23:21:25] (03PS1) 10Chad: WIP: make-release: Rewrite make-branches as branch [tools/release] - 10https://gerrit.wikimedia.org/r/399111 [23:21:50] Amir1: I see data there for other collectors on that same dashboard. Maybe something messed up about the ores scoring collector? [23:22:17] (03PS1) 10Chad: Remove weird cli protection from make-wmf-branch [tools/release] - 10https://gerrit.wikimedia.org/r/399112 [23:22:46] bd808: yeah, probably will look into it tomorrow or the day after [23:30:27] legoktm: hmm I see, thanks [23:30:57] PROBLEM - Puppet errors on deployment-kafka-jumbo-2 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [23:35:38] (03PS4) 10Legoktm: Allow _ in unit test method names [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/389668 (owner: 10Gergő Tisza) [23:35:42] (03CR) 10Legoktm: [C: 032] Allow _ in unit test method names [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/389668 (owner: 10Gergő Tisza) [23:36:11] PROBLEM - Puppet errors on deployment-netbox is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [23:37:25] PROBLEM - Puppet errors on deployment-kafka-jumbo-1 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [23:37:34] (03Merged) 10jenkins-bot: Allow _ in unit test method names [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/389668 (owner: 10Gergő Tisza) [23:38:01] 10Gerrit: Gerrit's ui does not correctly capitalise usernames in the git clone command - https://phabricator.wikimedia.org/T183205#3846779 (10Paladox) [23:38:59] 10Gerrit, 10Operations: Can't fetch from gerrit after updating ssh keys - https://phabricator.wikimedia.org/T183193#3846790 (10kaldari) 05Open>03Resolved a:03kaldari Turns out the username for gerrit has to be uppercase for some reason: ``` WMF1838:~ kaldari$ ssh -i .ssh/id_rsa.pub -p 29418 Kaldari@gerri... [23:39:14] (03CR) 10Legoktm: [C: 032] "Feel free to submit patches for the other URLs :)" [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/398945 (owner: 10Ricordisamoa) [23:41:18] 10Gerrit, 10Operations: Can't fetch from gerrit after updating ssh keys - https://phabricator.wikimedia.org/T183193#3846293 (10Paladox) The problem is when we set ldap lowercase earlier this year it made it lowercase through out the ui. Though they doint allow you to use different casing under ssh / git clone... [23:45:00] 10Gerrit, 10Upstream: Gerrit's ui does not correctly capitalise usernames in the git clone command - https://phabricator.wikimedia.org/T183205#3846806 (10Paladox) https://bugs.chromium.org/p/gerrit/issues/detail?id=8004 [23:45:12] (03PS2) 10Legoktm: Replace PEAR with Packagist in README.md link [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/398945 (owner: 10Ricordisamoa) [23:45:20] (03CR) 10Legoktm: [C: 032] Replace PEAR with Packagist in README.md link [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/398945 (owner: 10Ricordisamoa) [23:45:44] I guess I should switch it to Rebase-If-Necessary now [23:48:21] (03Merged) 10jenkins-bot: Replace PEAR with Packagist in README.md link [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/398945 (owner: 10Ricordisamoa)