IRC log of #schooltool for Wednesday, 2013-12-04

*** menesis has quit IRC01:11
*** th1a has quit IRC02:37
*** replaceafill has quit IRC03:21
*** basic` has quit IRC06:17
*** basic` has joined #schooltool06:18
*** basic` has joined #schooltool06:18
*** menesis has joined #schooltool07:52
*** menesis has quit IRC09:36
*** yvl has joined #schooltool09:40
*** testomania has joined #schooltool10:38
*** testomania has quit IRC10:39
*** mibofra- has joined #schooltool11:11
*** mibofra has quit IRC11:12
*** th1a has joined #schooltool12:26
*** menesis has joined #schooltool15:03
th1ahi menesis, yvl.15:30
yvlno replaceafill today?15:30
th1aI've been absorbing your spreadsheet.15:30
th1aHe should be along.15:30
th1aWhat are the color codes on sheet 3?15:31
yvlmy subjective evaluations of goodness15:32
yvlred - unusable15:32
yvlyellow - bad, but we can live with that15:32
yvlgreen - noticable, but usable15:32
yvlno colour - good15:32
*** replaceafill has joined #schooltool15:33
th1aI guess one reason there hasn't been much complaining is that the gradebook is pretty good.15:33
replaceafillgood afternoon15:33
yvlhey replaceafill15:33
replaceafillsorry i'm late (electricity problems... again...)15:33
yvlmakes sense, th1a15:34
th1aWhat is "MB passed through setstate"?15:34
yvlI'll try to explain quickly15:34
yvlbefore being loaded objects are in "ghost" stae15:34
yvljust id in database and a connection15:35
yvlwhen we access any information on object it gets activated via setstate15:35
yvltakes data from pickle cache, and passes to setstate to create the actual object15:36
yvlMB passed is persistence machinery estimate15:36
yvlactual data in memory will differ15:36
yvlbut it's a good random stat to look at15:36
*** replaceafill_ has joined #schooltool15:37
replaceafill_(this is f****** ridiculous...)15:37
replaceafill_sorry, continue guys :)15:37
yvlbtw, pickle cache keeps most of objects ready for deserialization15:37
*** replaceafill has quit IRC15:37
*** replaceafill_ is now known as replaceafill15:38
yvlso if object pickle stays in cache, creating it is faster15:38
yvlbut __setstate__ still takes it's toll15:38
yvland when there's many objects / lots of data...15:38
yvlI think it would be good to chat about this in a hangout at some point15:39
yvljust comment on some parts of the table, replaceafill might find it interesting :)15:40
th1aYes.  If replaceafill wasn't bouncing around, we could do it now.15:40
th1aBut it would be a good idea for him to understand the particulars.15:41
th1aWhat's your conclusion at this point, yvl?15:41
yvlwe should go through with latest optimizations15:41
yvlthey should hold fine15:41
yvlor give at least acceptable performance15:42
yvlnote that trunk is tested without temporal relationships15:42
th1aThat is "temporal"15:42
yvltemporal realtionships -> section enrollment with dates and so on15:42
yvlpre-enrolled student is realted to a section but may not in the section today15:43
yvl* may not be in15:43
yvlso that adds some extra weight, and would probably make plain implementation unusable in APS-sized DB15:43
yvlso if everybody's ok with it,15:44
yvlI'm going to clean up the code and push to trunk15:44
yvlit should take 1-2 days15:45
th1aAnd the global catalogs?15:45
yvlwill be there too15:45
replaceafillyvl, could you push the unclean code today so i can look around?15:45
replaceafilli mean, to a branch15:46
replaceafillyvl, not sure if you already explained this15:46
replaceafillbut what does hot means?15:46
replaceafillsecond load?15:47
th1aIs that essentially only the time for two consecutive loads of the same page?15:47
replaceafillafter just restarting the service?15:47
replaceafillah ok, got it15:47
th1aAs time goes on you don't know how fast it "cools."15:48
yvlit will also be the time if that page, or that data is used A LOT15:48
yvlusually it cools pretty quickly on low usage15:48
th1aWhich also makes it more important anyhow.15:48
yvlyes, optimizing cold loads is also important15:49
yvlnot just the first time after server restart15:49
yvlthere are some obvious blunders left15:49
yvllike go to School -> People15:49
yvland select Students group15:49
yvlit's twice as slow as just listing students group15:50
yvlshouldn't be15:50
yvlyou can probably see such blunder in person's home page15:51
yvlwhen it passes lots of date on second load, it's a sign that something stupid is happening15:52
replaceafilllots of  date?15:52
replaceafillsecond load should always be faster, right?15:53
yvlas a rule of thumb15:54
yvlalso, I've sent zipped profiling data15:55
yvlsudo apt-get install xdot15:56
yvlto view them15:56
yvlgraphs of time spent on calls15:56
yvlall roads lead to adapter_hook!15:56
yvlalso setstate15:57
yvloh, also15:58
yvlwhen you see setstate going through wait calls15:58
yvlit's talking with ZEO15:59
yvlthat it slo!15:59
yvlthat is slow15:59
replaceafillyvl, i've tests/trunk/open test-1-school-cold.gv16:00
yvlnext time it is easier to push the data through setstate16:00
replaceafillcould you explain it a little?16:00
replaceafilli mean16:00
replaceafillhow do you interpret the "chart?"16:00
yvltop left16:00
yvla red nod "interpret"16:00
yvlnot connected to it, a bit to the right16:01
yvl"run app"16:01
yvlrunapp is the full page load16:01
yvl14 seconds16:01
yvlinterpret leads to tal machinery16:01
replaceafillyou mean all the left part of the chart is related to templates?16:02
replaceafillto template rendering?16:02
yvltop-left blobish inter-connected thing16:02
yvlif you look at such graphs long enough, code flow patterns become familiar16:03
yvlnodes are methods16:03
yvlwith time spent in them16:03
yvlred nodes are noticibly worst16:04
yvlyellow and green are noticibly bad16:04
yvleverything else is blue16:04
yvlnext to each line, there's an amount of calls16:04
replaceafillso, __call__ in the middle calls _extractDelimiters 3 times16:05
replaceafill_extractOrientation 3 times16:05
replaceafilland so on?16:05
replaceafillgot it16:05
yvlfor example, you can easily find setstate here16:06
yvla red node16:06
yvlwith everything leading to it16:06
yvlleft - bottom16:06
yvlof graph16:06
replaceafill13005.31 ms16:06
yvlfrom 14 seconds, 13 were spend rebuilding objects16:07
yvland 7.5 pulling data through ZEO16:07
yvl(see load -> loadex -> call -> wait)16:07
yvlyou can clearly see 13.000 calls16:08
yvlso can imagine how fast it is to load an object on average16:08
*** replaceafill_ has joined #schooltool16:08
yvl<yvl> so16:09
yvl<yvl> from 14 seconds, 13 were spend rebuilding objects16:09
yvl<yvl> and 7.5 pulling data through ZEO16:09
yvl<yvl> (see load -> loadex -> call -> wait)16:09
yvl<yvl> you can clearly see 13.000 calls16:09
yvl<yvl> so can imagine how fast it is to load an object on average16:09
yvlyou can also see16:09
yvla big red not of getTargetsByRole16:10
*** replaceafill_ has quit IRC16:10
*** replaceafill_ has joined #schooltool16:10
replaceafill_dear lord!16:10
replaceafill_apologies guys16:10
yvlI'll just keep talking16:10
yvlplease open realtime irc logs :)16:11
*** replaceafill has quit IRC16:11
yvlbtw, if you follow the chain of callers from getRelatedObjects16:12
yvlyou'll see __len__ called a few times16:12
yvlthat iterates slowly and load everything!16:12
yvloh, also16:13
yvlif you zoom in close enough to a node16:13
yvlyou can move mouse around it16:13
yvlover the arrow links16:14
yvlsometimes they become red, and you can click to follow to next / prev node16:14
yvlthey're quirky though16:14
*** replaceafill has joined #schooltool16:15
*** replaceafill_ has quit IRC16:15
yvlwhat else...16:15
yvllook for areas with heavily interconnected nodes16:16
*** replaceafill_ has joined #schooltool16:16
*** replaceafill has quit IRC16:16
yvlthey usually have a logical purpose16:16
*** replaceafill_ is now known as replaceafill16:16
yvllike "template rendering"16:16
yvl"rendering a table"16:16
yvlalso, I put some text files with dumped object loads16:17
yvlbasically hijacked setstate16:17
yvlso if you look at load-1-school-cold.txt16:18
yvlin trunk and temporal316:18
yvlyou can see how 13k loads of Links16:19
yvlwere replaced with a fewer amount of BTree buckets16:19
yvland catalogs are made of indexes, those made of BTrees, btrees have lots of buckets16:19
yvlif you have any more questions, please don't hesitate to ask!16:20
*** menesis has quit IRC16:20
*** replaceafill has quit IRC16:21
* yvl done th1a 16:21
yvlbored people to death :/16:21
th1aso are you going to finish the catalogs after finishing the temporal relationships?16:21
*** menesis has joined #schooltool16:21
th1aAnd then after that you and replaceafill will be looking at just picking out bad coding on slow views?16:22
yvlwell, my latest code has both temporal relationships and catallogs16:22
yvlit would make sense to address the slow views after that16:22
yvland see if it runs faster on other real databases we can get access to16:23
yvland again16:24
yvlit's not that bad of a coding16:24
th1aI wasn't sure if you were talking about checking the two in separate steps.16:24
yvlbut there are places that follow "first make it work then make it fast" rule16:24
th1aAlso, 13,000 students is WAY more than SchoolTool is designed for.16:24
yvlI'll check those both in at the same time16:24
th1aSo it SHOULD be somewhat slow.16:24
yvlwith 3k students, a students group would take ~3 seconds to load16:25
th1aUsable, but it doesn't have to be sub-second response.16:25
th1aEsp. for admin pages.16:25
th1aIs there anything in particular we can do about student search?16:25
yvlas in16:26
yvlit is likely16:27
yvlhaven't checked what causes this exactly16:27
th1aThat would be a good one to work on.16:27
th1aOK, thanks yvl.16:28
menesisI have been working on intervention16:28
menesiswhere I left off16:28
menesisallow student view his own interventions. don't allow add/edit16:29
menesiswas never updated after permissions remap16:30
menesislooks ok now, will package today16:30
menesisread about stable release updates, needs a bug with a patch for each bugfix, tags, all reviewed16:32
menesis...or not, if it can qualify as micro release exception16:32
th1aAll reviewed by whom?16:32
menesisSRU reviewing team16:33
th1aHow far is that from what you've already done?16:33
menesisI have an upstream bugfix release16:34
menesisno new features16:34
th1aI mean, do you have the documentation, essentially?16:35
menesismade specifically to suit the requirements, so I hope I can push it through16:35
menesisbut I'll have to chat to someone to guide me through16:36
th1aOK.  It doesn't seem like more bugs are popping up.16:37
menesishave to write about regression potential, get someone else to verify that each bug is fixed...16:38
th1aOK... it is kind of up to you whether it is worth doing that instead of some of the other outstanding issues.16:39
menesisOK, I just read the documentation and not quite sure where to start16:39
th1aIt sounds like a headache.16:40
th1aSee if you can get some guidance.16:40
menesisand yes, I'd rather fix bugs, write docs, or clean code rather than go through bureaucracy16:41
menesisso permissions is all I worked on.16:42
* menesis done16:42
th1aOK thanks menesis.16:42
th1aI guess we've lost replaceafill.16:43
th1aIt looks like we might actually finally have the contract in St. Kitts we talked about... in the spring?16:44
th1areplaceafill is just going to have to extract their skills from a bunch of pdf's.16:45
th1aSo some of these disappearing clients from around the world are popping back up.16:45
th1aNext thing you know, we're going to hear from Cambodia!16:46
th1aThat's it.16:46
th1aThanks guys.16:46
th1aI'm sitting in the trough of my cold right now...16:46
th1aHave a good week/end.16:47
* th1a drops the bag of gravel.16:47
yvlget better th1a16:47
*** replaceafill has joined #schooltool16:55
*** replaceafill has joined #schooltool16:55
replaceafillsorry i missed the meeting16:56
replaceafillth1a, you still around?16:56
*** mibofra- is now known as mibofra17:05
*** mibofra has quit IRC17:05
*** mibofra has joined #schooltool17:05
*** veloutin has joined #schooltool18:40
*** veloutin has left #schooltool18:40
th1areplaceafill:  I am here.18:53
replaceafillhey th1a18:53
replaceafilli just wanted to say that i finished the levels work18:53
replaceafillfor courses18:53
th1aOK, good.18:53
replaceafillviews, importer/exporter and copy from previous year18:53
replaceafillwhat's next?18:54
replaceafilland i failed badly last night with Telly's pdf :(18:54
th1aAh... that's just what I was going to ask about.18:54
replaceafillyou were right, the unit tables are a mess18:54
th1aIt will make me feel better if it really isn't so easy.  ;-)18:55
th1aOCR-ing them is probably easier!18:55
replaceafillthat's what i was thinking18:55
replaceafilli remember i did that in the NGO i worked for18:55
replaceafilllike 6 years ago18:55
replaceafillto extract tables data18:55
replaceafillthe problem is, i don't remember at all how i did it :D18:56
th1aFrom pdfs?18:56
replaceafillscanned images18:56
replaceafillwhich are worst than the pdfs18:56
replaceafillconverting from pdf to image is really simple18:56
replaceafillit's the next step what i don't remember18:56
th1aI'd think the main thing is if there is a library that handles tables.18:58
replaceafillnot sure, but some of the tables don't look so "tablish" to me18:59
replaceafillbecause of the way the text is extracted18:59
replaceafillbut i don't know19:00
th1aOh, true.19:00
replaceafillthat's why i'd prefer the ocr approach first19:00
th1aOK, take a look at that.19:01
*** menesis has quit IRC19:49
*** menesis has joined #schooltool20:55

Generated by 2.15.1 by Marius Gedminas - find it at!