[Vol-dev] [patch] Pylint improvements for new_object

[Michael Cohen]

Hi Mike,
 Thanks for the patch. I tried to update it in the spirit of the new
object framework.
Im sure you already know what im about to say but this is to clarify
for other members of the list.

When you add a vtype overlay like

+    '_KUSER_SHARED_DATA' : [ None, { \
+    'SystemTime' : [ None, ['WinTimeStamp', {}]], \
+    'TimeZoneBias' : [ None, ['WinTimeStamp', {}]], \
+    }],

You are saying that a WinTimeStamp object should be used to decode
these fields. This overrides the standard definition in vtypes.py:

 '_KUSER_SHARED_DATA' : [ 0x338, { \
   'SystemTime' : [ 0x14, ['_KSYSTEM_TIME']], \
   'TimeZoneBias' : [ 0x20, ['_KSYSTEM_TIME']], \
   'SuiteMask' : [ 0x2d0, ['unsigned long']], \
   'NumberOfPhysicalPages' : [ 0x2e8, ['unsigned long']], \

so in your ident module when you do this:

k = object2.NewObject("_KUSER_SHARED_DATA",
win32.info.KUSER_SHARED_DATA, addr_space, profile=self.profile)

you will actually get an instance of WinTimeStamp back. Since adding
and subtracting time is quite common its worth adding a __sub__ method
to this class so you can just do:

return k.SystemTime - k.TimeZoneBias

Which is more readable and easier to understand - also note that this
should return a WinTimeStamp object - typically with the new command
reorg we try to ensure that we dont pass strings until the very end -
so as not to lose information. Note that expanding a WinTimeStamp into
a string will format it. You should try to use construct like
outfd.write("                Datetime: %s\n" % data['ImageDatetime'])

 instead of

outfd.write("                Datetime: " + data['ImageDatetime'])

The latter breaks when data is not a string (which it should not usually be)

Finally try to return NoneObject( reason) from functions rather than
None - this allows you to do stuff like:

       for task in win32.tasks.pslist(addr_space, self.profile):
           if task.Peb.CSDVersion:

instead of

+        for task in win32.tasks.pslist(addr_space, self.profile):
+            if task.Peb is not None:
+                if task.Peb.CSDVersion is not None:

Which is more error prone because you can forget to check for None at
a certain level for a timebomb bug. Admittadly I fixed a bug where
pslist() didnt do this it was my fault - but thats what we aim for.
Note that NoneObjects can be deferenced as much as you like to produce
the same NoneObject (i.e. the original fault is propagated).

This is important since in memory forensics sometime you will fail to
dereference a point which you thought was there (e.g. if its paged
out) so at least this allows the code to proceed without raising and
prevents you from having to check every single dereference for
success.

Hope this help,
Michael.

On Mon, Sep 21, 2009 at 8:34 AM, Mike Auty <mike.auty at gmail.com> wrote:
> Michael Cohen wrote:
>> Hi Mike,
>
> Hiya Michael!
>
>> Ok I will push it to the repository. I must admit im not a fan of svn
>> so im still a bit rusty in using it. Is there any way to get you
>> access to the new object branch? (this question is directed at project
>> admins btw).
>
> Thanks very much for committing that.  I know what you mean about svn, I
> tend to use git to track everything, and then it can just push back into
> svn when it needs too (and it'll remind you if you missed a file, just
> like hg!).  5;)
>
>> Ahh the linux tasks is really not even integrated at all yet. Its kind
>> of neglected since noone seems to be interested in linux atm :-(.
>
> I think Jon (echo6) might be interested in the Linux side of things.
> I've also got the old truecrypt plugin on look at on my todo list (way,
> way down on my todo list, but there at least), so we might be able to
> help bolster that.  It might be more worthwhile persuing the Mac OS
> X/BSD side of things though, given the distribution of machines people
> might run up against (particularly if people start looking at iPhones on
> ARM)...  5:)
>
>> heh - see here is my svn ignorance showing again - svn doesnt
>> automatically tell you you missed adding a file as far as i can see
>> (like hg does) - sorry I forgot to add that file.
>
> No problem, thanks for adding it back in.  5:)
>
>> The scan2 stuff is faster because its a more optimised scanning
>> technique - not really related to the new object framework. Basically
>> I was going to convert all the basic modules to the new framework as
>> well as we can. It will require a re-engineering of the module rather
>> than just a search/replace though since the new framework is much more
>> concise and readable - case in point is the example I gave previously
>> of _CM_HIVE.
>
> Thanks, I had a go at starting to convert over some of the modules.
> I've done datetime and ident for now, just to get me going.  Any chance
> you could take a look at them (part of the attached patch, since I
> touched other bits of the tree).  I'm still not fully up to speed on
> everything, so any criticisms would be greatly appreciated!  5:)
>
>> It will be clearer quite quickly which ones we need and which are not
>> so needed - e.g. the pstree one is basically pslist with some extra
>> stuff.  If you look at the memory_objects//Windows/xp_sp2.py you can
>> see that the new object classes are starting to implement their own
>> methods to do stuff they want - for example to iterate over an
>> _EPROCESS's handles you can just call:
>>
>> for h in process.handles():
>>     xxx
>>
>> this basically obsoletes Volatility/forensics/win32/handles.py
>
> That's very cool!  The next step I guess will be to start removing the
> older/unnecessary stuff to clear it all up.  5:)
>
>> Thats a great idea - although better than a single version string it
>> might be better to implement separate versions for each subsystem and
>> let the plugin decide if its ok to work with - e.g. new_object_version
>> 1.0 etc. Maybe just have a version in the command class?
>
> Yep, that seems reasonable.  The command class can keep a requirements
> list of features, and the framework can check it satisfies them all.
> That will allow for newer versions and new features, and to gracefully
> deprecate then remove old versions.
>
>> It might work this way im not sure - but the intension is to produce
>> better code, so I would suggest a better replacement for each plugin
>> should be written in the spirit of the new object design.
>
> Absolutely, as I say, I've started with the easy ones, but I have every
> intention to moving getting rid of vmodules (and then clearing out as
> much of the win32.* as are no longer necessary/use the old object model.
>
>> Oh i wrote a page about it once -
>> http://www.pyflag.net/cgi-bin/moin.cgi/Volatility_Object_Framework
>>
>> This is just me playing around and not any "official" way to generate
>> vtypes. It should probably be added to the repo.
>
> Brilliant, that's just the kind of documentation I was looking for.  It
> gives a good overview of what's going on.  Thanks!  5:)
>
>> Developement typically occured ad hoc and people
>> needed to essentially patch the framework as well as provide plugins
>> which sort of defeats the whole point of plugins. One of the problems
>> has been the inability to add complex vtype handling via plugins - a
>> problem which is addressed in the new branch.
>
> The overlay idea looks really good, but I'm still a little unclear how
> to integrate it cleanly (ie, how to have it affect just a single module,
> in case one overlays changes that break another)...
>
>> ## Maintained for backward compatibility do not use in new code
>
> Yeah, I saw that, it was just still used all over the place, and unless
> it goes away, I think people will keep using it...  5:(
>
>> I was a bit reluctant to completely remove old object support - but I
>> think I might need to do that to make it clear what files are going to
>> be removed.
>
> Yeah, I think so too, but it should make everything a lot easier to
> follow (and much easier to document!).  5:)
>
> Mike  5:)
>