[Vol-dev] [patch] Pylint improvements for new_object

[Michael Cohen]

Hi Mike,

On Sun, Sep 20, 2009 at 8:05 PM, Mike Auty <mike.auty at gmail.com> wrote:
> Hiya Michael,
> You're quite right, the patch I attached was entirely and completely
> non-existant (it was a bzip2 of an empty file).  Sorry about that
> everybody!  5:(  You should find the full thing attached now.

Ok I will push it to the repository. I must admit im not a fan of svn
so im still a bit rusty in using it. Is there any way to get you
access to the new object branch? (this question is directed at project
admins btw).

>> Not exactly - the new object scheme itself doesnt need these as these
>> functions were merged into the new object code. However, since I still
>> had the old object code lying around there might have been some
>> references left over.
>
> There were, I think it's just the one call to list_do in
> forensics.linux.tasks.

Ahh the linux tasks is really not even integrated at all yet. Its kind
of neglected since noone seems to be interested in linux atm :-(.

> It's not (at least, not the volatility svn tree), it's included in this
> giant patch though (as, unfortunately, is linked_list).  The only other
> thing that the code seemed to be missing before it would work correctly
> was a definition of _LDR_DATA_TABLE_ENTRY, which this patch also adds.

heh - see here is my svn ignorance showing again - svn doesnt
automatically tell you you missed adding a file as far as i can see
(like hg does) - sorry I forgot to add that file.

> I spotted pstree, it's the plugin I was going to use as the basis for
> modifying the others to using load_as!  5:)  Unfortunately, I've no idea
> which plugins duplicate functionality, and which simply do things a
> different way (all of the blahscan2 functions appear *much* quicker than

The scan2 stuff is faster because its a more optimised scanning
technique - not really related to the new object framework. Basically
I was going to convert all the basic modules to the new framework as
well as we can. It will require a re-engineering of the module rather
than just a search/replace though since the new framework is much more
concise and readable - case in point is the example I gave previously
of _CM_HIVE.

> the blahscan functions, but produce pretty much the same output), so I
> was going to convert them all and then ask people to help me cut away
> the ones we don't need...

It will be clearer quite quickly which ones we need and which are not
so needed - e.g. the pstree one is basically pslist with some extra
stuff.  If you look at the memory_objects//Windows/xp_sp2.py you can
see that the new object classes are starting to implement their own
methods to do stuff they want - for example to iterate over an
_EPROCESS's handles you can just call:

for h in process.handles():
    xxx

this basically obsoletes Volatility/forensics/win32/handles.py

So my point is that instead of having a plugin which retrieves all the
handles, or all the processes or whatever, this functionality can be
added to the object classes via object plugins and then anyone else
can just use this method off the relevant object. This will require a
better integrated design and avoid duplication in all the plugins.

> That sounds good to me, although it might be a courtesy to add some kind
> of a field in the forensics.commands.command object which specifies
> which version of volatility is needed for the plugin, just so that uses
> end up with a "this plugin won't work with this version" message, rather
> than a python stacktrace.  5;)  That'd need some planning to figure out
> though (since there may be times a version bumps without breaking
> compatibility).

Thats a great idea - although better than a single version string it
might be better to implement separate versions for each subsystem and
let the plugin decide if its ok to work with - e.g. new_object_version
1.0 etc. Maybe just have a version in the command class?

> Ok, I was more just asking about code conversion, since the function
> siganture for the Object constructor seemed similar to the call for
> NewObject, I was wondering if changing the code was going to be as easy
> as just changing all Objects to NewObjects.  It sounds like I'm going to
> have to know a lot more about how it works than I do at the moment to
> start converting them (which also means the pylint patch may break some
> things in a couple of places as I was doing the import clean-ups).

It might work this way im not sure - but the intension is to produce
better code, so I would suggest a better replacement for each plugin
should be written in the spirit of the new object design.

> There's still a few places where this is in use and is just reading a
> 'char', rather than a specifically designated type
> (linux.scan/win32.scan2).  I think that's what I was asking, will it be
> possible to fold in default types (like 'char', 'int', etc) into the
> NewObject model, so that we can effectively drop forensics.object, and
> then later rename forensics.object2 to its rightful place.  5:)

Well these types are also new objects too - there is no difference
between an int and an _EPROCESS - they are both just objects you can
instantiate at any AS offset you want - the point is that usually, you
would be better off instantiating the entire struct at the offset,
rather than each element using int, char etc. Its more readable what
you are doing and also more maintainable. The reason why many of the
old plugins had to read specific ints etc by hand was because the
vtypes system lacked the ability to define complex ways of reading
data automatically (case in point is that variable sized string i
mentioned in an earlier post). Hopefully we can avoid the need to
resort to that now.

> Ah, yeah, I was wondering where the autogenerator code is (or are you
> just saying it could be autogenerated from .h files?  Presumably that
> should be somewhere in the repository?  How does it decide which classes
> to yoink from the .h, or does it grab all of them and you weed them out
> manually?

Oh i wrote a page about it once -
http://www.pyflag.net/cgi-bin/moin.cgi/Volatility_Object_Framework

This is just me playing around and not any "official" way to generate
vtypes. It should probably be added to the repo.

> Well, it will become difficult, but other large projects do manage
> plugins (although granted, developers often have more time to work on
> them that our userbase does).  This comes back to the compatibility flag
> in the plugin.  As long as it can safely be discarded if it doesn't work
> properly, that should be fine.  Bringing modules into the tree is great,
> but only if there's someone there to maintain them, otherwise you're
> just burdening the developers with access to the tree with more code to
> maintain.  That's why planning a really good interface for everything,
> and then trying hard not to change it, is a worthwhile challenge...

Yeah I agree - the flip side is that you dont want to stop development
because you might break plugins. If people want to keep their plugins
separate thats fine, but they need to maintain them. Hopefully the new
obeject framework will be stable enough to keep things in check. Yeah
version strings are ideal but they only work with stable release
cycles (as in one can advertise their plugin works with volatiltiy 1.3
or 1.4 or whatever) - we have not had enough releases yet to see how
that works out. Developement typically occured ad hoc and people
needed to essentially patch the framework as well as provide plugins
which sort of defeats the whole point of plugins. One of the problems
has been the inability to add complex vtype handling via plugins - a
problem which is addressed in the new branch.

> Well, I'm having a go at it, but I think it's mostly the mix of the two
> systems that's hindering adoption.  Not knowing whether you can or can't
> use read_obj, etc and then whether you can use forensics.addrspace.  The
> documentation will certainly help, particularly if it shows how to
> properly use the latest system for everything, though...  5;)

So read_obj is obsolete - also forensics.addrspace is also obsolete as
the comment suggests:

## Maintained for backward compatibility do not use in new code
class FileAddressSpace:

I was a bit reluctant to completely remove old object support - but I
think I might need to do that to make it clear what files are going to
be removed.

Thanks,
Michael.