2008-08-17 00:00 <flips> now lets see if printk has it
2008-08-17 00:00 <shapor> set it up where?
2008-08-17 00:00 <shapor> @tux3.org?
2008-08-17 00:01 <shapor> if you move it you'd have re-subscribe all the members
2008-08-17 00:01 <shapor> sounds like a pita
2008-08-17 00:03 <flips> true
2008-08-17 00:03 <flips> printk has %t
2008-08-17 00:03 <flips> so maybe we should use it on the use every feature principle
2008-08-17 00:04 <shapor> whats %t?
2008-08-17 00:05 <flips> difference of pointers :p
2008-08-17 00:05 <shapor> does printf?
2008-08-17 00:06 <flips> also
2008-08-17 00:09 <flips> http://lxr.linux.no/linux+v2.6.26.2/lib/vsprintf.c#L779
2008-08-17 00:09 -!- flips(~phillips@phunq.net) has left #tux3
2008-08-17 00:09 -!- flips(~phillips@phunq.net) has joined #tux3
2008-08-17 00:10 <flips> http://lxr.linux.no/linux+v2.6.26.2/lib/vsprintf.c#L779 <- the documentation for printk
2008-08-17 00:11 <shapor> ah i see
2008-08-17 00:11 <shapor> yes we should be using that
2008-08-17 00:11 <shapor> perfect
2008-08-17 00:13 <flips> finally
2008-08-17 00:13 <shapor> i've always wanted to read the printf man page to see all the features i never use
2008-08-17 00:13 <shapor> this is a good reason to do so
2008-08-17 00:14 <flips> uhoh
2008-08-17 00:15 <shapor> ?
2008-08-17 00:16 <shapor> nice no compile warnings on 64
2008-08-17 00:17 <shapor> %ti does make more sense than %tu
2008-08-17 00:17 <shapor> in case of an error it will be easy to see the negative value then
2008-08-17 00:18 <flips> right
2008-08-17 03:06 -!- pgquiles(~pgquiles@132.Red-217-125-199.dynamicIP.rima-tde.net) has joined #tux3
2008-08-17 06:38 <flips> morning pgquiles
2008-08-17 06:39 <pgquiles> hey flips
2008-08-17 06:40 <flips> a little early for here
2008-08-17 06:44 <pgquiles> what time is it over there?
2008-08-17 06:44 <flips> 6:45
2008-08-17 06:44 <pgquiles> that hurts :-)
2008-08-17 06:45 <flips> yup
2008-08-17 06:45 <flips> gotta get some more zzz's
2008-08-17 06:45 <flips> see all the tux3 checkins
2008-08-17 06:45 <flips> shapor got another patch, 3rd or 4th I think
2008-08-17 06:46 <pgquiles> did a Jos van den Oever write you?
2008-08-17 06:46 <pgquiles> he's the guy developing Strigi
2008-08-17 06:52 <flips> not yet
2008-08-17 06:53 <pgquiles> he'll eventually do
2008-08-17 06:54 <flips> getting indexing working well for kde would please me
2008-08-17 06:55 <pgquiles> me too :-)
2008-08-17 06:55 <pgquiles> anyway, strigi is not tied to kde
2008-08-17 06:56 <flips> even better
2008-08-17 06:56 <flips> but kde will use it early I suppose
2008-08-17 06:56 <pgquiles> yes, it is already using it
2008-08-17 06:57 <pgquiles> in 4.0 and 4.1 you need to explicitly enable it but in 4.2 will be enabled by default
2008-08-17 06:57 <flips> 
ACTION loves kde

2008-08-17 06:57 <pgquiles> flips: there's Camp KDE in January in Jamaica, you know :-)
2008-08-17 06:57 <flips> wow, even more south than I already am
2008-08-17 06:58 <flips> I will keep it in mind
2008-08-17 06:59 <pgquiles> I'm going to sleep one hour, I still need to recover from aKademy
2008-08-17 06:59 <pgquiles> see you later
2008-08-17 06:59 <flips> bye
2008-08-17 11:56 -!- pgquiles_(~pgquiles@126.Red-80-39-172.dynamicIP.rima-tde.net) has joined #tux3
2008-08-17 13:42 <flips> 
ACTION goes wacks tree_expand into 2 big pieces

2008-08-17 13:43 <flips> typos galor
2008-08-17 13:54 <flips> whacking completed
2008-08-17 13:54 <flips> now to make it make sense
2008-08-17 13:58 <flips> 7 parameters for insert_child now down to 6...
2008-08-17 15:14 <flips> iattr.c coming soon
2008-08-17 15:14 <shapor> you think perhaps a dir should contain a default file attr entry?
2008-08-17 15:15 <flips> yes for sure
2008-08-17 15:15 <shapor> in most cases dirs contain very similar files
2008-08-17 15:15 <flips> and be inherited instead of storing the same attr in each inode
2008-08-17 15:15 <shapor> exactly
2008-08-17 15:15 <flips> inode attrs come in groups
2008-08-17 15:15 <flips> one of the groups is ctime/mode/uid/guid
2008-08-17 15:15 <flips> the "create" group
2008-08-17 15:16 <flips> that one can usually be inherited except for ctime
2008-08-17 15:16 <flips> well
2008-08-17 15:16 <flips> so, break out the ctime
2008-08-17 15:16 <flips> so there is maybe two flavors of the create group
2008-08-17 15:16 <flips> ctime only and ctime plus ownership
2008-08-17 15:17 <flips> it might be ok to always separate them
2008-08-17 15:17 <flips> a version and a 4 bit attr type field have to go in each attr
2008-08-17 15:18 <flips> so: struct ctime { u64 kind:4, version:10, time: 50 };
2008-08-17 15:20 <flips> and struct owner { u64 kind:4, version:10, mode: 50, uid:32, gid:32 };
2008-08-17 15:20 <flips> 8 bytes for the first, 16 bytes for the second
2008-08-17 15:21 <flips> 16 bytes saved every time it is possible to inherit the owner from the directory
2008-08-17 15:21 <flips> which is something like 99.9% of the time on my system I think
2008-08-17 15:25 <shapor> 
ACTION just wrote a sick one-liner to determine the average number of different file perms a directory contains

2008-08-17 15:26 <shapor> here is the distribution from my web server:
2008-08-17 15:26 <shapor>     651 0
2008-08-17 15:26 <shapor>    4318 1
2008-08-17 15:26 <shapor>      92 2
2008-08-17 15:26 <shapor>       8 3
2008-08-17 15:26 <shapor>       4 4
2008-08-17 15:27 <shapor> so, 651 directories with no files in them, 4318 with only one set of perms on the files contained in it
2008-08-17 15:27 <shapor> 92 with 2, etc
2008-08-17 15:27 <shapor> aka, output of:
2008-08-17 15:27 <shapor> find / -xdev -type d -exec sh -c 'ls -l $1 | awk "/^\-/ {print \$1}" | sort -u |wc -l' {} {} \; | sort | uniq -c
2008-08-17 15:38 <shapor> across the entire filesystem there are only 30 unique permissions settings
2008-08-17 15:38 <shapor> only about 500k files
2008-08-17 15:38 <flips> nice observation
2008-08-17 15:38 <shapor> s/only/on/
2008-08-17 15:38 <flips> so there should be owner atoms?
2008-08-17 15:38 <flips> or sorry
2008-08-17 15:38 <flips> permission atoms
2008-08-17 15:40 <flips> above, 4318 out of 6000 files save 16 bytes
2008-08-17 15:41 <flips> running on my system
2008-08-17 15:41 <shapor> those are directories
2008-08-17 15:41 <shapor> not files
2008-08-17 15:41 <flips> right
2008-08-17 15:41 <flips> directories have attrs just like files, I'm getting a sense of overall saving
2008-08-17 15:42 <shapor> ah
2008-08-17 15:42 <flips> considering just files, what is the single permissions percentage?
2008-08-17 15:43 <shapor> you mean, how many files live in directories with only one set of permissions?
2008-08-17 15:43 <shapor> this is also a fun one to run:
2008-08-17 15:43 <shapor> find / -xdev -printf "%m\n" 2>/dev/null| sort | uniq -c| sort -n 
2008-08-17 15:44 <shapor> out of 53972 files, only 24 unique permissions sets
2008-08-17 15:44 <shapor> 41605 files are mode 644
2008-08-17 15:46 <flips> talking]
2008-08-17 15:49 <shapor> 39537 files live in directories with only one set of permissions
2008-08-17 15:52 <shapor> its total brain damage to store 0644 all over the filesystem
2008-08-17 15:58 <shapor> 
ACTION keeps getting more excited about tux3

2008-08-17 16:01 <shapor> flips: where did "16 bytes" come from?
2008-08-17 16:02 <shapor> each set of permissions is only 12 bits
2008-08-17 16:02 <flips> struct owner { u64 kind:4, version:10, mode: 50, uid:32, gid:32 };
2008-08-17 16:02 <flips> they come in groups
2008-08-17 16:02 <shapor> oh
2008-08-17 16:02 <shapor> i see
2008-08-17 16:02 <flips> they have to be discriminated for traversal and versioned
2008-08-17 16:02 <flips> that is why putting them in groups is a win
2008-08-17 16:03 <shapor> mode:50 ?
2008-08-17 16:03 <flips> some extra ;-)
2008-08-17 16:03 <flips> at this point I am trying to preserve 8 byte granularity
2008-08-17 16:03 <flips> I am not sure if that matters
2008-08-17 16:03 <shapor> i see
2008-08-17 16:03 <flips> I can go to byte aligned and save more space
2008-08-17 16:03 <flips> probably the right thing to do actually
2008-08-17 16:04 <flips> but it is a detail of iattr.c
2008-08-17 16:04 <shapor> thats a lot of extra mode bits
2008-08-17 16:04 <shapor> why?
2008-08-17 16:04 <flips> struct owner { u64 kind:4, version:10, mode: 16, pad: 36, uid:32, gid:32 };
2008-08-17 16:04 <flips> ok?
2008-08-17 16:05 <shapor> oh i guess we should support "chattr"
2008-08-17 16:05 <flips> struct owner { u64 kind:4, version:10, mode:16, pad:4, uid:32, gid:32 }; <- 12 bytes]
2008-08-17 16:05 <flips> um
2008-08-17 16:05 <flips> I'm on drugs
2008-08-17 16:06 <flips> 16 bytes
2008-08-17 16:06 <flips> so you were right, :50 was just silly
2008-08-17 16:06 <flips> chattr must be supported of course
2008-08-17 16:06 <shapor> how many bits does chattr need
2008-08-17 16:06 <flips> chattr the command?
2008-08-17 16:07 <shapor> yeah
2008-08-17 16:07 <shapor> ioctl(3, EXT2_IOC_GETFLAGS, 0x7fff16bffeec) = 0
2008-08-17 16:08 <flips> struct owner { u64 kind:4, version:10, mode:20, uid:32, gid:32 }; <- 12 bytes]
2008-08-17 16:08 <flips> 20 mode bits should do it
2008-08-17 16:08 <shapor> we need to support that interface
2008-08-17 16:09 <shapor> how does that work with the vfs layer
2008-08-17 16:09 <flips> http://lxr.linux.no/linux+v2.6.26.2/include/linux/ext2_fs.h#L200
2008-08-17 16:09 <flips> http://lxr.linux.no/linux+v2.6.26.2/+ident=11795060
2008-08-17 16:11 <flips> ext2 has 22 of its own per file flags, mostly bullshit
2008-08-17 16:11 <flips> unused
2008-08-17 16:11 <flips> things like compression
2008-08-17 16:12 <flips> tail packing
2008-08-17 16:12 <flips> #define EXT2_INDEX_FL                   FS_INDEX_FL     /* hash-indexed directory */ <- one of the few that actually got implemented (by me)
2008-08-17 16:12 <shapor> heh
2008-08-17 16:13 <flips> anyway, we will divide them into commonly used flags that go in the 20 mode bits and rare ones that go in some other attribute type
2008-08-17 16:14 <flips> 16 basic attribute types should be enough
2008-08-17 16:14 <flips> one of those types is "extended attribute"
2008-08-17 16:14 <shapor> ok
2008-08-17 16:14 <shapor> well there are only 8 spare mode bits
2008-08-17 16:15 <shapor> not counting chattr
2008-08-17 16:15 <shapor> i was just trying to figure out how many chattr needed
2008-08-17 16:15 <flips> 
ACTION does man chattr

2008-08-17 16:15 <flips> not very many I think
2008-08-17 16:16 <shapor> ----------------- /tmp
2008-08-17 16:16 <shapor> thats a lot of dashes (output from lsattr)
2008-08-17 16:17 <flips> 98% of your files could inherit struct owner from parent directory
2008-08-17 16:17 <flips> just crunched it
2008-08-17 16:17 <flips> my system is:
2008-08-17 16:17 <flips>    4113 0
2008-08-17 16:17 <flips>   16326 1
2008-08-17 16:17 <flips>     419 2
2008-08-17 16:17 <flips>      20 3
2008-08-17 16:17 <flips>       8 4
2008-08-17 16:17 <flips>       3 5
2008-08-17 16:17 <flips>       1 7
2008-08-17 16:18 <shapor> yeah
2008-08-17 16:18 <flips> my system is 97.3
2008-08-17 16:19 <flips> %
2008-08-17 16:19 <shapor> probably rarely under 90%
2008-08-17 16:19 <flips> you are 97.7 actually
2008-08-17 16:19 <shapor> even on multiuser systems
2008-08-17 16:19 <flips> you are 97.6 actually
2008-08-17 16:19 <flips> true
2008-08-17 16:19 <shapor> its rare to work in the same directory as another user
2008-08-17 16:20 <flips> that is 261K of kernel cache saved for full stat
2008-08-17 16:20 <flips> on my system
2008-08-17 16:20 <flips> which is not particularly big fs
2008-08-17 16:21 <shapor> man ls -lR is fucking slow
2008-08-17 16:21 <flips> ddtree is fscking fast ;-)
2008-08-17 16:22 <shapor> it stats /etc/localtime for EVERY file ??
2008-08-17 16:22 <flips> bleah
2008-08-17 16:22 <flips> work of the fsf unfortunately
2008-08-17 16:27 <flips> http://lxr.linux.no/linux+v2.6.26.2/fs/ext2/inode.c#L1147 <- ext2_set_inode_flags
2008-08-17 16:28 <flips> total of 10 flags
2008-08-17 16:28 <flips> per inode
2008-08-17 16:28 <flips> almost all seem sensible
2008-08-17 16:31 <flips> struct owner { u64 kind:4, version:10, mode:18, uid:32, gid:32 }; <- 12 bytes
2008-08-17 16:31 <flips> too tight maybe
2008-08-17 16:32 <flips> unless mode is an atom
2008-08-17 16:32 <flips> anyway bask to inums
2008-08-17 20:38 -!- tux3bot(~tux3bot@yzf.shapor.com) has joined #tux3
2008-08-17 20:41 -!- shapor(~shapor@yzf.shapor.com) has joined #tux3
2008-08-17 21:38 <flips> tuxkey_t *next_key(struct treepath *path, int levels)
2008-08-17 21:38 <flips> {
2008-08-17 21:38 <flips> 	for (int level = levels; --level < 0; )
2008-08-17 21:38 <flips> 		if (!finished_level(path, level))
2008-08-17 21:38 <flips> 			return &path[level].next->key;
2008-08-17 21:38 <flips> 	return NULL;
2008-08-17 21:38 <flips> }
2008-08-17 21:39 <flips> shapor, how does it look/
2008-08-17 21:40 <shapor> ok?
2008-08-17 21:42 <shapor> flips: that doesn't seem very complex
2008-08-17 21:43 <flips> just looks up the path until it finds a level where we haven't read all the way to the end of the index block yet
2008-08-17 21:44 <flips> there we find a key that separates the subtree we are in (a leaf) from the next subtree to the right
2008-08-17 21:45 <shapor> yeah i was able to read the c, you dont need to put that in a comment ;)
2008-08-17 21:45 <shapor> just wondering why you were pasting it
2008-08-17 21:45 <flips> aw, I was just doing that ;-)
2008-08-17 21:45 <flips> because I haven't tested it
2008-08-17 21:45 <shapor> heh
2008-08-17 21:45 <flips> it's part of the inode table block insertion stuff
2008-08-17 21:46 <shapor> oh
2008-08-17 21:46 <shapor> i dont know how that all fits together
2008-08-17 21:46 <flips> knowing the successor key tells us whether we should advance to the next block to search it or insert a new one, becauase the successor block is too far away in key space
2008-08-17 21:46 <shapor> i think you described it in a post recently
2008-08-17 21:46 <shapor> or no
2008-08-17 21:48 <flips> not in that detail
2008-08-17 21:48 <flips> and not everybody will be able to see the purpose from the code, so I put in the comment
2008-08-17 21:53 <flips> this function the way it is written can actually replace a nasty little bit of code in the delete, because it returns a pointer
2008-08-17 21:53 <flips> instead of the key
2008-08-17 21:53 <flips> in the delete, we have to find that successor key and change it
2008-08-17 21:57 <shapor> i was thinking about the size and mtime
2008-08-17 21:57 <shapor> do we have to update those?
2008-08-17 21:57 <flips> yes
2008-08-17 21:57 <flips> every time the file changes
2008-08-17 21:57 <flips> with a little bit of slop
2008-08-17 21:57 <flips> definitely on every fsync
2008-08-17 21:58 <flips> this is a job for the log of course
2008-08-17 21:58 <flips> tuxkey_t *p = next_key(path, levels), next = p ? *p : MAX_INODES; <- obscure enough?
2008-08-17 21:59 <shapor> no i like it
2008-08-17 21:59 <shapor> makes perfect sense
2008-08-17 22:00 <flips> traditionalists have  trouble with C code like that
2008-08-17 22:00 <flips> then they need to be reminded that C99 was last millenium already
2008-08-17 22:00 <flips> doesn't work though
2008-08-17 22:01 <flips> a lot of folks are still scared of structure assignment
2008-08-17 22:01 <flips> let along inline decls
2008-08-17 22:01 <flips> let alone
2008-08-17 22:01 <shapor> what doesnt work
2008-08-17 22:01 <flips> it doesn't work to remind them
2008-08-17 22:01 <shapor> oh
2008-08-17 22:01 <flips> traditionalists are like that
2008-08-17 22:02 <flips> it is convenient that the maximum inodes and maximum blocks of a filesystem are the same
2008-08-17 22:02 <flips> that means there is a density of one inode per block
2008-08-17 22:02 <shapor> howso
2008-08-17 22:03 <flips> which is convenient for deciding which inum to use
2008-08-17 22:03 <flips> over time with deletes that nice simple relationship deteriorates
2008-08-17 22:03 <flips> then we need other mojo
2008-08-17 23:04 <shapor> flips: why are some of the data structures in the c files and some in tux3.h?
2008-08-17 23:13 <flips> because me being lame
2008-08-17 23:13 <flips> the c files are included
2008-08-17 23:13 <flips> like header files
2008-08-17 23:13 <flips> this is only for while it's in heavy hack mode
2008-08-17 23:14 <flips> when everything has to be declared twice it gets in the way of quick refactoring
2008-08-17 23:21 <shapor> so struct inode is just from ext2 i guess?
2008-08-17 23:21 <flips> vfs
2008-08-17 23:22 <shapor> with version added ?
2008-08-17 23:22 <flips> generic inode has version, yes, it is a different, bogus version
2008-08-17 23:22 <flips> stupid misnamed field
2008-08-17 23:22 <shapor> i see
2008-08-17 23:23 <flips> it is really a change counter
2008-08-17 23:23 <shapor> ah
2008-08-17 23:23 <flips> so that ext2/3 knows when it has to go look in the directory in case the entry it was positioned on was deleted
2008-08-17 23:23 <flips> it is that specific
2008-08-17 23:23 <flips> calling that a version was a heinous crime
2008-08-17 23:28 <shapor> i'll just ignore it ;)
2008-08-17 23:29 <flips> it gets used in dir.c for revalidate, nowhere else
2008-08-17 23:36 <shapor> causes running inode.c to fail
2008-08-17 23:36 <flips> what does?
2008-08-17 23:39 <shapor> er i thought i pasted the assert
2008-08-17 23:40 <shapor> whoops
2008-08-17 23:40 <shapor> [10869] probe: Failed assertion "(ops->leaf_sniff)(sb, buffer->data)"
2008-08-17 23:41 <flips> that's fresh out of repository?
2008-08-17 23:42 <flips> maybe you have to give a filename on the command line?
2008-08-17 23:42 <shapor> ah
2008-08-17 23:42 <flips> not a good error message to be sure
2008-08-17 23:42 <shapor> yeah, my bad
2008-08-17 23:43 <flips> woo, it really tries hard even when it has no backing file
2008-08-17 23:44 <flips> I wonder if this is good
2008-08-17 23:45 <flips> generated 32 bitmap blocks, that is not good behaviou
2008-08-17 23:45 <flips> behavior
2008-08-17 23:46 <flips> [18006] main: Bitmap flush failed (Bad file descriptor) <- this is nice
2008-08-17 23:47 <flips> fd '(null)' = -1 (0xffffffff bytes) <- this maybe not so nice
2008-08-17 23:49 <flips> sb->image.blocks has a wierd number when fd is -1
2008-08-17 23:51 <flips> it's strange that printf doesn't support signed hexadecimal
2008-08-17 23:54 <flips> ok, fdsize64 now returns zero when it fails, better than overloading the size with -1, or maybe not
2008-08-17 23:54 <flips> anyway, inode.c then proceeds to fail on the leaf
2008-08-17 23:54 <flips> I wonder if those failure paths are decent
2008-08-17 23:55 <flips> probe needs to fail and not think it got a leaf
2008-08-17 23:55 <shapor> hrm signed hex
2008-08-17 23:56 <flips> makes sense to me
2008-08-17 23:56 <shapor> i suppose
2008-08-17 23:57 <shapor> why doesn't fdsize64 just take a u64*
2008-08-17 23:57 <flips> it should
2008-08-17 23:57 <flips> and return err
2008-08-17 23:57 <flips> I thought I made it do that
2008-08-17 23:57 <flips> but the patch probably got danked
2008-08-17 23:57 <shapor> like the ioctl does