tag:blogger.com,1999:blog-4276649639758349933.post2432728254548138650..comments2010-07-14T17:14:08.535+01:00Comments on GSOC2010. BioPerl Alignment Subsystem Refactoring: An open question of how to be memory efficientJun Yinhttp://www.blogger.com/profile/01246427171492799531noreply@blogger.comBlogger1125tag:blogger.com,1999:blog-4276649639758349933.post-76371607288248024032010-06-07T21:03:01.865+01:002010-06-07T21:03:01.865+01:00So, this is something that will have to scale to p...So, this is something that will have to scale to possibly millions of sequences. <br /><br />First: the 'database' backend should be abstracted out of the equation, used to just grab the proper sequences and so on needed for the work. In this way you can implement this backend however you want, just give it consistent methods (interface). The default should probably be an in-memory store of some kind, whereas others can be tied to a simple DB via DBI, etc, are lazy, etc. Start with something that works (in-memory) and work out from there. You can wrap particular tools (Bio::Samtools, Bio::BigFile, etc) as needed. Or use the already set up tools Mark Jensen and others have worked on if it fills the need.<br /><br />Storable will just serialize the data, correct? So one would use it in conjunction with DB_File or others. Tie::File is a possibility maybe for a lazy parser, not sure how well it scales to very large files.pyrimidinehttps://www.blogger.com/profile/13078227774973006844noreply@blogger.com