IRC log of #schooltool for Tuesday, 2013-11-12

*** menesis has quit IRC00:27
*** menesis has joined #schooltool00:28
*** menesis has quit IRC03:35
*** yvl has joined #schooltool10:36
*** menesis has joined #schooltool11:21
*** menesis has quit IRC12:37
*** menesis has joined #schooltool12:46
*** menesis has quit IRC12:56
*** khildin has joined #schooltool13:46
*** menesis has joined #schooltool14:51
*** replaceafill has joined #schooltool15:23
*** th1a has joined #schooltool15:30
th1ahi replaceafill.15:31
replaceafillhey th1a15:31
th1aOK... where are we replaceafill?15:34
replaceafillwith the social skills profiling?15:35
replaceafilli was able to cut from ~900 seconds to ~200 seconds15:35
replaceafill(which is still a lot)15:35
th1aHow'd you do that?15:36
th1aYeah... three minutes!15:36
replaceafillthere were two unnecessary loops15:37
replaceafillexecuted too many times15:37
replaceafilli just moved them out15:38
replaceafillone of them was in the cando gradebook15:38
replaceafillthe main problem in the social skills case15:38
replaceafillis that they're using 4 terms15:38
replaceafilland the gradebook usually lookup previous sections15:39
replaceafilli now understand what yvl says about performance in relationships15:39
th1aWhich is?15:40
replaceafillthe lookup i mention uses .findAllEquivalent()15:40
replaceafillwhich in turn uses relationship to check equivalent skills15:40
replaceafillthose line show higher cumtime in the profile output15:41
replaceafillit's also worth mentioning that this single view gathers ~470 skills15:42
replaceafilljust for a single student15:42
th1aSo yvl is working on indexing for relationships, right?15:43
th1aHi yvl.  ;-)15:43
replaceafillhey yvl!15:43
yvlhey :)15:43
th1aIs there a way we can try this with the view replaceafill is working on, soonish?15:45
replaceafillhere's my latest profile results:15:45
yvlsoonish is probably the right word :)15:45
replaceafillmaybe yvl can spot something obvious :)15:45
replaceafillth1a, at this point i'd like to try using a table formatter15:47
th1aWell, I guess the question is whether we should switch to some small tasks for the bugfix release.15:48
th1aFor a couple days.15:48
replaceafillbut i guess that won't improve the previous score lookup anyway15:48
replaceafilltrying yvl's change sounds like a good idea15:49
yvl586907 calls to getRelationshipLinks :)15:49
replaceafillyvl, yes :(15:49
th1aWhat are these looking for?15:51
th1aA higher previous score?15:51
replaceafillany previous score != None15:52
yvlhmm, cando gradebook getScore is taking a long time15:52
th1aWhat does it do with these?15:52
* th1a needs a reminder...15:53
yvlfinds skills that are equivalent to this skill15:53
th1aYeah... why do we need them in this view?15:54
replaceafillsorry, back15:54
replaceafillth1a, equivalents are used to get the "same" skill in the previous section15:55
th1aYeah... why?15:55
th1aIs it doing some kind of calculation?15:56
replaceafillno, it's just for presenting it in the gradebook15:56
* th1a is trying to remember this...15:57
th1aI guess I should be looking at it.15:57
th1aDoes it always show previous terms?15:57
replaceafillif the score is None15:57
replaceafilland there are linked previous sections15:57
th1aOK, it is only looking if the current score is none.15:58
th1aOK, that's all I was trying to get at.15:58
th1aI didn't get your pseudocode, replaceafill.15:59
replaceafillah sorry :D15:59
replaceafillsince skillsets are global16:00
replaceafilland they're deployed using the original __name__16:00
replaceafillmaybe we could use __name__'s for the section skillset and skill16:00
replaceafillto get the equivalent in the previous sections16:01
replaceafillinstead of the relationship lookup16:01
replaceafillluckily, cando has a stest for this "feature"16:02
replaceafillso i'd probably can try this quickly16:02
yvlbtw, if you disable get previous score  (return None)16:04
yvleverything works fast enough?16:04
th1aThat's a good question.16:05
replaceafillon it!16:06
replaceafill626447 function calls (602976 primitive calls) in 0.759 seconds16:07
th1aWhat's that?16:07
replaceafillfull output:
replaceafillth1a, time with get previous score disabled16:07
th1aIt goes from .8 seconds to 200?16:08
yvlwhat happens if all grades are filled in?16:08
yvland maybe even have score history?16:09
yvl(edited twice, at least some of them)16:09
replaceafillah ok, will grade some16:09
yvlthanks man16:09
replaceafill(this is the profiling art yvl talked about) :D16:09
th1aOh, .8 is just finding nothing?16:10
yvlnot searching16:11
yvlthere's many equivalent skills there it seems16:11
th1aYeah, ok.16:11
yvlwe have 1000+ score history requests that result in 550000+ relationship requests16:12
th1a500 each?16:12
th1aDo we know why so many?16:14
yvlbut if for some reason we are interested, we can use pdb in a single findAllEquivalent call and see16:15
th1aYou'd think we could cut that down A LOT.16:16
replaceafill         442136 function calls (419167 primitive calls) in 0.397 seconds16:17
replaceafillwith most of the skills graded16:17
replaceafillfor that section16:17
replaceafill(and search disabled)16:17
replaceafillgoing to enable search16:17
replaceafill45323163 function calls (44878668 primitive calls) in 54.711 seconds16:19
replaceafill(that's with search enabled)16:19
replaceafilland ~80% of the skills graded16:19
yvland previous was 200 secs, right?16:19
replaceafillbecause ALL the grades are searched16:20
th1aALL of what exactly?16:20
replaceafillth1a, i mean, this forth term section doesn't have any grades of its own16:21
yvlhow many equivalent skills on average are we talking here?16:21
th1aShouldn't it be, like, four?16:23
replaceafilli don't think so16:23
replaceafillfindAllEquivalent gets skills from any section16:23
replaceafilli mean, not only linked one16:23
th1aIs this a section view?16:24
replaceafilla student gradebook view16:24
replaceafillth1a, remember that view with checkmarks and X's16:25
replaceafillthe autism center uses16:25
th1aIs this the report view or the entry view?16:25
replaceafillthe entry16:25
replaceafillreport does basically the same16:26
th1aSo... there is really no reason we're looking at all sections in this case?16:26
th1aSo let's not do that!16:26
replaceafillthat's the reason of filterEquivalent16:26
replaceafillit filters the skills16:27
replaceafillbut it acts AFTER the lookup16:27
th1aOK, lets NOT do that?16:27
replaceafillyvl, do the __name__ lookup seems fragile to you?16:27
th1aIs this where an index helps?16:27
yvla little bit yes, replaceafill16:29
replaceafillyvl, kk16:29
yvlnot saying that would not work though16:29
* yvl didn't think about it yet :)16:30
yvlin any case, the right thing to do here is not to use findallequivelent obviously16:32
yvlyou need findEquivalent(activity, section)16:33
yvland need it to work fast :)16:33
yvlif you'll end up using __name__s, so be it :)16:33
yvlbut probably you'll end up simply checking16:35
yvlfor skill in section something16:35
yvlif skill in activity.equivalent16:35
yvlof equivalent = set(activity.equivalent)16:36
yvland then the for16:36
yvland then if skill in equivalent16:36
yvlsomething along the lines16:36
replaceafillso basically redefining findAllEquivalent to filter by section?16:37
replaceafillbtw, i just pdb'd getPreviousScore16:37
replaceafillit finds 475 raw equivalent for a single skill16:37
th1aIf we can find the score for the current term in essentially no time at all, why can't we do the same for the previous linked terms?16:38
replaceafilland after filtering only gets a single skill back16:38
yvlok, so if it makes sense to ALWAYS look only in previous sections16:39
yvl(not projects, not antything else)16:39
yvlyou should override getPreviousScore16:39
yvland instead of using .findAllEquivalent and the .filterEquivalent16:40
yvluse your new .finEquivalentInSection(activity, section)16:40
replaceafillgot it16:40
th1aProjects in the relevant sections?16:41
yvlth1a, we'd need to profile further16:41
yvlbut my bet is that loading all 475 skills is slowing things a bit16:41
yvlwell 1025 * 47516:41
yvltechnically, a student could have completed the equivalent skill not just in previous section16:42
yvlbut in a "previous" section taught by another instructor16:42
th1aThat's not relevant to this view.16:43
yvlso it's still an equivalent skill, still a section in previous term, but not a directly previous section with this instructor16:43
yvlthat's good for us then :)16:44
th1aWe're really only interested in linked terms for this view.16:48
th1aEspecially since it is the edit view.16:50
th1aOK, I guess you should keep working on this replaceafill... although I'd like that extra leading space bug to be gone in 2.6.116:51
replaceafillth1a, oh16:51
replaceafilli tested it yesterday16:51
replaceafilli didn't see the "extra leading" part16:51
th1aMaybe it is a Chrome thing.16:51
replaceafilli know how to do the value.strip()16:51
th1aYes, indeed.16:52
th1aDo that.16:52
replaceafilland i had a question for yvl16:52
replaceafillif you're still around :)16:52
yvlI am :)16:52
replaceafillit's about the entity part16:53
replaceafilli remember in old times we did:16:53
replaceafill<!ENTITY nbsp    "&#160;">16:53
replaceafillat the top of the rml16:53
replaceafilli haven't checked yet, but do you think this problem could be related to that tag missing in the current templates16:54
replaceafilli'm not sure what's the role of <!entity ...> in the rml16:54
yvlIt's a good question if we want non breaking spaces at all here16:57
yvlI'd rather have them stripped16:57
replaceafillbut that's discrimination against old old people :P16:57
yvloh, right16:58
yvlalso me16:58
yvlStripping - yes16:58
th1aI don't mind losing double spaces, if that's the question.16:58
replaceafillah cool16:58
replaceafilli could strip and remove multiple spaces in the same data converter16:59
replaceafillfor the htmlfragmentfield widget16:59
replaceafilli guess i need to use a new problem... i mean, a regular expression17:00
replaceafillunless there's an easier way to remove multiple spaces in strings :)17:00
th1aI'd be happy to just strip the leading spaces and move onto the next issue.17:02
th1aOK, I have to go get the girls.17:03
th1aLater guys.17:03
replaceafillthanks th1a17:03
yvlbye th1a17:06
yvlreplaceafill, it may be just a missing entity in rml17:06
replaceafillyvl, right17:06
replaceafilli'll check17:06
yvlpound is probably missing too17:07
replaceafilli noticed pound17:07
yvlI'd say we need that in all templates17:07
replaceafillwhy do we need it?17:07
replaceafilli thought it was pound, as in british pound?17:07
yvlyes, I think fckn editor escapes it17:08
yvlif you could check, that would be great :)17:08
yvloh, and entity should be somewhere in schooltool/skin/flourish/rml/17:09
replaceafillit was in rml_macros17:09 probably17:09
replaceafillyvl, correct, the fck editor saves it as &#160;17:18
replaceafillu'<p>&#160;Foo bar.&#160; Another quick.</p>'17:18
yvlthat editor!17:18
replaceafillthis is a side effect17:24
replaceafillof the FCKConfig.ProcessHTMLEntities = false change17:24
replaceafillthe change was needed because the editor saves ó or ñ characters as entities17:25
replaceafillwhich in turn are displayed like that in the pdfs17:25
replaceafillbut in that case the extra spaces are kept17:25
replaceafillu'<p> Foo bar.  Another quick fix. This is ni&ntilde;o, comunicaci&oacute;n.</p>'17:25
* replaceafill goes look up the fckeditor docs17:26
*** khildin has quit IRC17:34
*** khildin has joined #schooltool17:49
replaceafillturns out that just setting:18:24
replaceafill-FCKConfig.ProcessHTMLEntities = false ;18:24
replaceafill+FCKConfig.IncludeLatinEntities = false ;18:24
replaceafill+FCKConfig.IncludeGreekEntities = false ;18:24
replaceafillfixes all the problems with the editor :|18:24
replaceafilláéí... characters work18:24
replaceafilland spaces are not escaped anymore18:25
*** menesis has quit IRC18:59
*** menesis has joined #schooltool19:10
*** menesis has quit IRC19:59
*** khildin has quit IRC20:43
*** khildin has joined #schooltool20:51
*** menesis has joined #schooltool20:52
*** khildin has quit IRC20:59
*** khildin has joined #schooltool21:25
*** khildin has quit IRC21:46
*** khildin has joined #schooltool21:49
replaceafillfrom 900 to 10 seconds :)23:44
replaceafillhopefully same output :P23:45
th1aWhat did you do?23:45
replaceafillrunning cando tests now23:45
th1aThat's still a long time, ofc...23:45
replaceafillused recursion properly i think23:45
th1aSo were we just looping 1000x what we needed to?23:46
replaceafillthere are still some numbers that seem high in the profile output23:46
th1aYeah... sometimes these things seem more like bugs than performance constraints.23:46
replaceafilli stopped lookup after finding the first equivalent too23:46
th1aOh... so?  I'm not sure what that means.23:47
replaceafillfindAllEquivalent was finding... well... all!23:47
replaceafilland after filtering23:47
replaceafillyou always got 1 skill23:47
replaceafilli'm still using for loops to traverse as yvl suggested23:47
replaceafillbut they stop after the match is found23:48
replaceafillthe previous sections calculation was done over and over23:48
replaceafillfinding the same 4 sections for each worksheet iteration23:49
replaceafilland the method didn't needed the 423:49
replaceafilljust the previous one23:49
th1aWell, I'm glad this is more a bug than just hitting the limits of the ZODB or something like that.23:54

Generated by 2.15.1 by Marius Gedminas - find it at!