| *** menesis has quit IRC | 01:11 | |
| *** th1a has quit IRC | 02:37 | |
| *** replaceafill has quit IRC | 03:21 | |
| *** basic` has quit IRC | 06:17 | |
| *** basic` has joined #schooltool | 06:18 | |
| *** basic` has joined #schooltool | 06:18 | |
| *** menesis has joined #schooltool | 07:52 | |
| *** menesis has quit IRC | 09:36 | |
| *** yvl has joined #schooltool | 09:40 | |
| *** testomania has joined #schooltool | 10:38 | |
| *** testomania has quit IRC | 10:39 | |
| *** mibofra- has joined #schooltool | 11:11 | |
| *** mibofra has quit IRC | 11:12 | |
| *** th1a has joined #schooltool | 12:26 | |
| *** menesis has joined #schooltool | 15:03 | |
| th1a | hi menesis, yvl. | 15:30 |
|---|---|---|
| yvl | hello | 15:30 |
| yvl | no replaceafill today? | 15:30 |
| th1a | I've been absorbing your spreadsheet. | 15:30 |
| th1a | He should be along. | 15:30 |
| menesis | hi | 15:31 |
| th1a | What are the color codes on sheet 3? | 15:31 |
| yvl | umm | 15:32 |
| yvl | my subjective evaluations of goodness | 15:32 |
| yvl | red - unusable | 15:32 |
| yvl | yellow - bad, but we can live with that | 15:32 |
| yvl | green - noticable, but usable | 15:32 |
| yvl | no colour - good | 15:32 |
| *** replaceafill has joined #schooltool | 15:33 | |
| th1a | I guess one reason there hasn't been much complaining is that the gradebook is pretty good. | 15:33 |
| replaceafill | good afternoon | 15:33 |
| yvl | hey replaceafill | 15:33 |
| replaceafill | sorry i'm late (electricity problems... again...) | 15:33 |
| yvl | makes sense, th1a | 15:34 |
| th1a | What is "MB passed through setstate"? | 15:34 |
| yvl | I'll try to explain quickly | 15:34 |
| yvl | before being loaded objects are in "ghost" stae | 15:34 |
| yvl | state | 15:34 |
| yvl | just id in database and a connection | 15:35 |
| yvl | when we access any information on object it gets activated via setstate | 15:35 |
| yvl | takes data from pickle cache, and passes to setstate to create the actual object | 15:36 |
| yvl | MB passed is persistence machinery estimate | 15:36 |
| yvl | actual data in memory will differ | 15:36 |
| th1a | OK. | 15:36 |
| yvl | but it's a good random stat to look at | 15:36 |
| *** replaceafill_ has joined #schooltool | 15:37 | |
| th1a | Yeah. | 15:37 |
| replaceafill_ | (this is f****** ridiculous...) | 15:37 |
| replaceafill_ | sorry, continue guys :) | 15:37 |
| yvl | btw, pickle cache keeps most of objects ready for deserialization | 15:37 |
| *** replaceafill has quit IRC | 15:37 | |
| *** replaceafill_ is now known as replaceafill | 15:38 | |
| yvl | so if object pickle stays in cache, creating it is faster | 15:38 |
| yvl | but __setstate__ still takes it's toll | 15:38 |
| yvl | and when there's many objects / lots of data... | 15:38 |
| yvl | I think it would be good to chat about this in a hangout at some point | 15:39 |
| yvl | just comment on some parts of the table, replaceafill might find it interesting :) | 15:40 |
| th1a | Yes. If replaceafill wasn't bouncing around, we could do it now. | 15:40 |
| th1a | But it would be a good idea for him to understand the particulars. | 15:41 |
| th1a | What's your conclusion at this point, yvl? | 15:41 |
| yvl | right | 15:41 |
| yvl | we should go through with latest optimizations | 15:41 |
| yvl | they should hold fine | 15:41 |
| yvl | or give at least acceptable performance | 15:42 |
| yvl | note that trunk is tested without temporal relationships | 15:42 |
| th1a | That is "temporal" | 15:42 |
| yvl | temporal realtionships -> section enrollment with dates and so on | 15:42 |
| yvl | pre-enrolled student is realted to a section but may not in the section today | 15:43 |
| th1a | Yes. | 15:43 |
| yvl | * may not be in | 15:43 |
| yvl | so that adds some extra weight, and would probably make plain implementation unusable in APS-sized DB | 15:43 |
| yvl | so if everybody's ok with it, | 15:44 |
| yvl | I'm going to clean up the code and push to trunk | 15:44 |
| yvl | it should take 1-2 days | 15:45 |
| th1a | OK. | 15:45 |
| th1a | And the global catalogs? | 15:45 |
| yvl | will be there too | 15:45 |
| replaceafill | yvl, could you push the unclean code today so i can look around? | 15:45 |
| replaceafill | i mean, to a branch | 15:46 |
| yvl | sure | 15:46 |
| replaceafill | thanks | 15:46 |
| replaceafill | yvl, not sure if you already explained this | 15:46 |
| replaceafill | but what does hot means? | 15:46 |
| replaceafill | second load? | 15:47 |
| yvl | yes | 15:47 |
| th1a | Is that essentially only the time for two consecutive loads of the same page? | 15:47 |
| yvl | yes | 15:47 |
| replaceafill | after just restarting the service? | 15:47 |
| yvl | yes | 15:47 |
| replaceafill | ah ok, got it | 15:47 |
| th1a | As time goes on you don't know how fast it "cools." | 15:48 |
| yvl | true | 15:48 |
| yvl | it will also be the time if that page, or that data is used A LOT | 15:48 |
| th1a | Right. | 15:48 |
| yvl | usually it cools pretty quickly on low usage | 15:48 |
| th1a | Which also makes it more important anyhow. | 15:48 |
| yvl | yes, optimizing cold loads is also important | 15:49 |
| yvl | not just the first time after server restart | 15:49 |
| yvl | there are some obvious blunders left | 15:49 |
| yvl | like go to School -> People | 15:49 |
| yvl | and select Students group | 15:49 |
| yvl | it's twice as slow as just listing students group | 15:50 |
| yvl | shouldn't be | 15:50 |
| yvl | you can probably see such blunder in person's home page | 15:51 |
| th1a | Ah. | 15:51 |
| yvl | when it passes lots of date on second load, it's a sign that something stupid is happening | 15:52 |
| replaceafill | lots of date? | 15:52 |
| yvl | dat | 15:52 |
| yvl | a | 15:52 |
| yvl | :D | 15:52 |
| replaceafill | ah | 15:53 |
| replaceafill | second load should always be faster, right? | 15:53 |
| yvl | yes | 15:54 |
| yvl | as a rule of thumb | 15:54 |
| yvl | oh | 15:55 |
| yvl | also, I've sent zipped profiling data | 15:55 |
| yvl | sudo apt-get install xdot | 15:56 |
| yvl | to view them | 15:56 |
| yvl | graphs of time spent on calls | 15:56 |
| yvl | all roads lead to adapter_hook! | 15:56 |
| yvl | also setstate | 15:57 |
| yvl | oh, also | 15:58 |
| yvl | when you see setstate going through wait calls | 15:58 |
| yvl | it's talking with ZEO | 15:59 |
| yvl | that it slo! | 15:59 |
| yvl | that is slow | 15:59 |
| th1a | Ah. | 16:00 |
| replaceafill | yvl, i've tests/trunk/open test-1-school-cold.gv | 16:00 |
| yvl | next time it is easier to push the data through setstate | 16:00 |
| replaceafill | could you explain it a little? | 16:00 |
| replaceafill | i mean | 16:00 |
| yvl | sure | 16:00 |
| replaceafill | how do you interpret the "chart?" | 16:00 |
| yvl | top left | 16:00 |
| yvl | a red nod "interpret" | 16:00 |
| replaceafill | yes | 16:01 |
| yvl | not connected to it, a bit to the right | 16:01 |
| yvl | "run app" | 16:01 |
| yvl | runapp is the full page load | 16:01 |
| yvl | 14 seconds | 16:01 |
| yvl | interpret leads to tal machinery | 16:01 |
| replaceafill | you mean all the left part of the chart is related to templates? | 16:02 |
| replaceafill | to template rendering? | 16:02 |
| yvl | top-left blobish inter-connected thing | 16:02 |
| yvl | yes | 16:02 |
| replaceafill | !!! | 16:03 |
| yvl | if you look at such graphs long enough, code flow patterns become familiar | 16:03 |
| yvl | nodes are methods | 16:03 |
| yvl | with time spent in them | 16:03 |
| yvl | red nodes are noticibly worst | 16:04 |
| yvl | yellow and green are noticibly bad | 16:04 |
| yvl | everything else is blue | 16:04 |
| replaceafill | ah | 16:04 |
| yvl | next to each line, there's an amount of calls | 16:04 |
| replaceafill | so, __call__ in the middle calls _extractDelimiters 3 times | 16:05 |
| replaceafill | _extractOrientation 3 times | 16:05 |
| replaceafill | and so on? | 16:05 |
| yvl | yes | 16:05 |
| replaceafill | got it | 16:05 |
| yvl | for example, you can easily find setstate here | 16:06 |
| yvl | a red node | 16:06 |
| yvl | with everything leading to it | 16:06 |
| yvl | left - bottom | 16:06 |
| yvl | of graph | 16:06 |
| replaceafill | 13005.31 ms | 16:06 |
| yvl | so | 16:07 |
| yvl | from 14 seconds, 13 were spend rebuilding objects | 16:07 |
| yvl | and 7.5 pulling data through ZEO | 16:07 |
| yvl | (see load -> loadex -> call -> wait) | 16:07 |
| yvl | you can clearly see 13.000 calls | 16:08 |
| yvl | so can imagine how fast it is to load an object on average | 16:08 |
| *** replaceafill_ has joined #schooltool | 16:08 | |
| replaceafill_ | :( | 16:09 |
| yvl | :/ | 16:09 |
| yvl | <yvl> so | 16:09 |
| yvl | <yvl> from 14 seconds, 13 were spend rebuilding objects | 16:09 |
| yvl | <yvl> and 7.5 pulling data through ZEO | 16:09 |
| yvl | <yvl> (see load -> loadex -> call -> wait) | 16:09 |
| yvl | <yvl> you can clearly see 13.000 calls | 16:09 |
| yvl | <yvl> so can imagine how fast it is to load an object on average | 16:09 |
| yvl | you can also see | 16:09 |
| yvl | a big red not of getTargetsByRole | 16:10 |
| *** replaceafill_ has quit IRC | 16:10 | |
| *** replaceafill_ has joined #schooltool | 16:10 | |
| replaceafill_ | dear lord! | 16:10 |
| replaceafill_ | apologies guys | 16:10 |
| yvl | right | 16:10 |
| yvl | I'll just keep talking | 16:10 |
| yvl | please open realtime irc logs :) | 16:11 |
| *** replaceafill has quit IRC | 16:11 | |
| yvl | http://schooltool.pov.lt/irclogs/%23schooltool.2013-12-04.log | 16:11 |
| yvl | btw, if you follow the chain of callers from getRelatedObjects | 16:12 |
| yvl | you'll see __len__ called a few times | 16:12 |
| yvl | that iterates slowly and load everything! | 16:12 |
| yvl | oh, also | 16:13 |
| yvl | if you zoom in close enough to a node | 16:13 |
| yvl | you can move mouse around it | 16:13 |
| yvl | over the arrow links | 16:14 |
| yvl | sometimes they become red, and you can click to follow to next / prev node | 16:14 |
| yvl | they're quirky though | 16:14 |
| *** replaceafill has joined #schooltool | 16:15 | |
| *** replaceafill_ has quit IRC | 16:15 | |
| yvl | what else... | 16:15 |
| yvl | look for areas with heavily interconnected nodes | 16:16 |
| *** replaceafill_ has joined #schooltool | 16:16 | |
| *** replaceafill has quit IRC | 16:16 | |
| yvl | they usually have a logical purpose | 16:16 |
| *** replaceafill_ is now known as replaceafill | 16:16 | |
| yvl | like "template rendering" | 16:16 |
| yvl | "rendering a table" | 16:16 |
| yvl | also, I put some text files with dumped object loads | 16:17 |
| yvl | basically hijacked setstate | 16:17 |
| yvl | so if you look at load-1-school-cold.txt | 16:18 |
| yvl | in trunk and temporal3 | 16:18 |
| yvl | you can see how 13k loads of Links | 16:19 |
| yvl | were replaced with a fewer amount of BTree buckets | 16:19 |
| yvl | and catalogs are made of indexes, those made of BTrees, btrees have lots of buckets | 16:19 |
| yvl | ok | 16:20 |
| yvl | if you have any more questions, please don't hesitate to ask! | 16:20 |
| *** menesis has quit IRC | 16:20 | |
| *** replaceafill has quit IRC | 16:21 | |
| * yvl done th1a | 16:21 | |
| th1a | lol | 16:21 |
| th1a | OK, | 16:21 |
| yvl | bored people to death :/ | 16:21 |
| th1a | so are you going to finish the catalogs after finishing the temporal relationships? | 16:21 |
| *** menesis has joined #schooltool | 16:21 | |
| th1a | And then after that you and replaceafill will be looking at just picking out bad coding on slow views? | 16:22 |
| yvl | well, my latest code has both temporal relationships and catallogs | 16:22 |
| yvl | it would make sense to address the slow views after that | 16:22 |
| yvl | and see if it runs faster on other real databases we can get access to | 16:23 |
| yvl | and again | 16:24 |
| yvl | it's not that bad of a coding | 16:24 |
| th1a | I wasn't sure if you were talking about checking the two in separate steps. | 16:24 |
| yvl | but there are places that follow "first make it work then make it fast" rule | 16:24 |
| th1a | Yes. | 16:24 |
| th1a | Also, 13,000 students is WAY more than SchoolTool is designed for. | 16:24 |
| yvl | I'll check those both in at the same time | 16:24 |
| th1a | So it SHOULD be somewhat slow. | 16:24 |
| yvl | true | 16:25 |
| yvl | with 3k students, a students group would take ~3 seconds to load | 16:25 |
| th1a | Usable, but it doesn't have to be sub-second response. | 16:25 |
| th1a | Esp. for admin pages. | 16:25 |
| yvl | yes | 16:25 |
| th1a | Is there anything in particular we can do about student search? | 16:25 |
| yvl | probably | 16:26 |
| yvl | as in | 16:26 |
| yvl | it is likely | 16:27 |
| th1a | OK. | 16:27 |
| yvl | haven't checked what causes this exactly | 16:27 |
| th1a | That would be a good one to work on. | 16:27 |
| yvl | right | 16:27 |
| th1a | OK, thanks yvl. | 16:28 |
| th1a | menesis? | 16:28 |
| menesis | I have been working on intervention | 16:28 |
| menesis | where I left off | 16:28 |
| menesis | allow student view his own interventions. don't allow add/edit | 16:29 |
| menesis | was never updated after permissions remap | 16:30 |
| menesis | looks ok now, will package today | 16:30 |
| menesis | read about stable release updates, needs a bug with a patch for each bugfix, tags, all reviewed | 16:32 |
| menesis | ...or not, if it can qualify as micro release exception | 16:32 |
| th1a | All reviewed by whom? | 16:32 |
| menesis | SRU reviewing team | 16:33 |
| th1a | How far is that from what you've already done? | 16:33 |
| menesis | I have an upstream bugfix release | 16:34 |
| menesis | no new features | 16:34 |
| th1a | I mean, do you have the documentation, essentially? | 16:35 |
| menesis | made specifically to suit the requirements, so I hope I can push it through | 16:35 |
| menesis | but I'll have to chat to someone to guide me through | 16:36 |
| th1a | OK. It doesn't seem like more bugs are popping up. | 16:37 |
| menesis | have to write about regression potential, get someone else to verify that each bug is fixed... | 16:38 |
| th1a | OK... it is kind of up to you whether it is worth doing that instead of some of the other outstanding issues. | 16:39 |
| menesis | OK, I just read the documentation and not quite sure where to start | 16:39 |
| th1a | It sounds like a headache. | 16:40 |
| th1a | See if you can get some guidance. | 16:40 |
| menesis | and yes, I'd rather fix bugs, write docs, or clean code rather than go through bureaucracy | 16:41 |
| menesis | OK. | 16:41 |
| th1a | OK. | 16:41 |
| menesis | so permissions is all I worked on. | 16:42 |
| * menesis done | 16:42 | |
| th1a | OK thanks menesis. | 16:42 |
| th1a | I guess we've lost replaceafill. | 16:43 |
| th1a | It looks like we might actually finally have the contract in St. Kitts we talked about... in the spring? | 16:44 |
| th1a | replaceafill is just going to have to extract their skills from a bunch of pdf's. | 16:45 |
| th1a | So some of these disappearing clients from around the world are popping back up. | 16:45 |
| th1a | Next thing you know, we're going to hear from Cambodia! | 16:46 |
| th1a | OK. | 16:46 |
| th1a | That's it. | 16:46 |
| th1a | Thanks guys. | 16:46 |
| th1a | I'm sitting in the trough of my cold right now... | 16:46 |
| th1a | Have a good week/end. | 16:47 |
| * th1a drops the bag of gravel. | 16:47 | |
| yvl | thanks | 16:47 |
| yvl | get better th1a | 16:47 |
| *** replaceafill has joined #schooltool | 16:55 | |
| *** replaceafill has joined #schooltool | 16:55 | |
| replaceafill | sorry i missed the meeting | 16:56 |
| replaceafill | th1a, you still around? | 16:56 |
| *** mibofra- is now known as mibofra | 17:05 | |
| *** mibofra has quit IRC | 17:05 | |
| *** mibofra has joined #schooltool | 17:05 | |
| *** veloutin has joined #schooltool | 18:40 | |
| *** veloutin has left #schooltool | 18:40 | |
| th1a | replaceafill: I am here. | 18:53 |
| replaceafill | hey th1a | 18:53 |
| replaceafill | i just wanted to say that i finished the levels work | 18:53 |
| replaceafill | for courses | 18:53 |
| th1a | OK, good. | 18:53 |
| replaceafill | views, importer/exporter and copy from previous year | 18:53 |
| replaceafill | what's next? | 18:54 |
| replaceafill | and i failed badly last night with Telly's pdf :( | 18:54 |
| th1a | Ah... that's just what I was going to ask about. | 18:54 |
| replaceafill | you were right, the unit tables are a mess | 18:54 |
| th1a | It will make me feel better if it really isn't so easy. ;-) | 18:55 |
| replaceafill | :D | 18:55 |
| th1a | OCR-ing them is probably easier! | 18:55 |
| replaceafill | that's what i was thinking | 18:55 |
| replaceafill | i remember i did that in the NGO i worked for | 18:55 |
| replaceafill | like 6 years ago | 18:55 |
| replaceafill | to extract tables data | 18:55 |
| replaceafill | the problem is, i don't remember at all how i did it :D | 18:56 |
| th1a | Hm... | 18:56 |
| th1a | From pdfs? | 18:56 |
| replaceafill | scanned images | 18:56 |
| replaceafill | which are worst than the pdfs | 18:56 |
| replaceafill | converting from pdf to image is really simple | 18:56 |
| replaceafill | it's the next step what i don't remember | 18:56 |
| th1a | I'd think the main thing is if there is a library that handles tables. | 18:58 |
| replaceafill | not sure, but some of the tables don't look so "tablish" to me | 18:59 |
| replaceafill | because of the way the text is extracted | 18:59 |
| replaceafill | but i don't know | 19:00 |
| th1a | Oh, true. | 19:00 |
| replaceafill | that's why i'd prefer the ocr approach first | 19:00 |
| th1a | OK, take a look at that. | 19:01 |
| *** menesis has quit IRC | 19:49 | |
| *** menesis has joined #schooltool | 20:55 | |
Generated by irclog2html.py 2.15.1 by Marius Gedminas - find it at mg.pov.lt!