Harvard ABCD Forge > Projects > Screensaver HTS LIMS > Tracker > Screensaver Tasks

[#1177] library import performance

Submitted By: Andrew Tolopko
Start Date* Feb 02 2007	End Date* Mar 02 2007
Priority: Medium High	Assignee: Andrew Tolopko Sean Erickson
Status: Closed	Component: N/A
Branch: iccbl	Impact: performance
Stakeholders:	RTID: 61221
Revision Found:	Resolution: Fixed/Implemented
Active: Inactive	Estimated Effort: Medium (<= week)
Planned Release : 1.8.0	Goals Addressed: Scalability
Components: Library Batch I/O	Priority Approved: Jan 13 2009
Audience:
Summary library import performance
Details performance is now pretty sucky, and some performance tuning would probably help a lot. a JProbe license would help! I can get a trial license if we don't have an official license by then. And some help from Andrew in how he got JProbe running for him would also be nice ;-)
Details performance is now pretty sucky, and some performance tuning would probably help a lot. a JProbe license would help! I can get a trial license if we don't have an official license by then. And some help from Andrew in how he got JProbe running for him would also be nice ;-)

Follow-ups
Submitted By: Andrew Tolopko Adddate: 2008-10-02 12:40:23 > [js163 - Fri Feb 02 10:22:30 2007]: > > performance is now pretty sucky, and some performance tuning would > probably help a lot. a JProbe license would help! I can get a trial > license if we don't have an official license by then. And some help from > Andrew in how he got JProbe running for him would also be nice ;-) JProbe is nice, but JProfiler is also an option. I think JProfiler is 1/3 the price and I was quite happy with it at my last job; very intuitive and had all the functionality I wanted. One thing I didn't like about JProbe was that memory and cpu profiling were handled by separate applications. Also JProbe's config paradigm left me confused--there are apparently 2 levels of config (app settings vs profiler session settings???), and I was always tripping over which one I was editing and which files were being updated when I edited settings. Re-launching from eclipse often caused an entirely new config to be created, instead of reusing an old config. I can't remember the details, but I know I didn't like it! JProbe installation, however, was fairly straightforward--after an initial hiccup that prevented it from running (some env config problem, which I can't remember now). After unzipping the distro, and running jpprofiler, I was able to use a menu option to add the eclipse integration hooks. The only real advantage of the eclipse integration, AFAIK, is that the classpath is automatically set. But as mentioned, it seems to ignore profiler config options you set in previous invocations. For now, I'd actually recommend trying JProfiler, to get an opinion on it, so that we can make a decision about which to buy.
Submitted By: Andrew Tolopko Adddate: 2008-10-02 12:40:23 Cool! I will give JProfiler an evaluation while I use it to profile the library import performance! -s Andrew N Tolopko via RT wrote: >> [js163 - Fri Feb 02 10:22:30 2007]: >> >> performance is now pretty sucky, and some performance tuning would >> probably help a lot. a JProbe license would help! I can get a trial >> license if we don't have an official license by then. And some help from >> Andrew in how he got JProbe running for him would also be nice ;-) > > JProbe is nice, but JProfiler is also an option. I think JProfiler is > 1/3 the price and I was quite happy with it at my last job; very > intuitive and had all the functionality I wanted. One thing I didn't > like about JProbe was that memory and cpu profiling were handled by > separate applications. Also JProbe's config paradigm left me > confused--there are apparently 2 levels of config (app settings vs > profiler session settings???), and I was always tripping over which one > I was editing and which files were being updated when I edited settings. > Re-launching from eclipse often caused an entirely new config to be > created, instead of reusing an old config. I can't remember the > details, but I know I didn't like it! > > JProbe installation, however, was fairly straightforward--after an > initial hiccup that prevented it from running (some env config problem, > which I can't remember now). After unzipping the distro, and running > jpprofiler, I was able to use a menu option to add the eclipse > integration hooks. The only real advantage of the eclipse integration, > AFAIK, is that the classpath is automatically set. But as mentioned, it > seems to ignore profiler config options you set in previous invocations. > > For now, I'd actually recommend trying JProfiler, to get an opinion on > it, so that we can make a decision about which to buy.
Submitted By: Andrew Tolopko Adddate: 2008-10-02 12:40:24 FWIW: compound library loading: 40 hours (~250K wells processed) rnai library loading: 1 hr 10 min Ran over the weekend, starting Fri 5pm (noting that import speed is probably limited by Pubchem I/O).
Submitted By: Andrew Tolopko Adddate: 2008-10-02 12:40:24 I have a suspicion that the library imports might run faster on your machine than mine as well! -s Andrew N Tolopko via RT wrote: > FWIW: > compound library loading: 40 hours (~250K wells processed) > rnai library loading: 1 hr 10 min > > Ran over the weekend, starting Fri 5pm (noting that import speed is > probably limited by Pubchem I/O).
Submitted By: Andrew Tolopko Adddate: 2008-10-02 12:40:25 John Sullivan via RT writes: > ID: 61221 > Subject: Re: [HMS #61221] library import performance > Status: open > Owner: Nobody in particular [Nobody] > > Requestors: john_sullivan@hms.harvard.edu > Cc: andrew_tolopko@hms.harvard.edu > Queue: ICCB-L:Informatics > > Ticket URL: https://rt.med.harvard.edu/Ticket/Display.html?id=61221 > > Comment & Take: mailto:take-61221@rt.med.harvard.edu > Reply & Resolve: mailto:resolve-61221@rt.med.harvard.edu > > This is correspondence. It is sent to all ticket watchers. > ------------------------------------------------------------------------- > I have a suspicion that the library imports might run faster on your > machine than mine as well! -s Well, FWIW: 3Ghz Intel D820 dual core 160GB 7200RPM SATA disk 1GB RAM Gentoo Linux, custom-compiled postgres 8.1.4 (?) Comcast 6Mb/s IPV4 cxn @ > Andrew N Tolopko via RT wrote: > > FWIW: > > compound library loading: 40 hours (~250K wells processed) > > rnai library loading: 1 hr 10 min > > > > Ran over the weekend, starting Fri 5pm (noting that import speed is > > probably limited by Pubchem I/O). >
Submitted By: Andrew Tolopko Adddate: 2008-10-02 12:40:25 would go higher than 1 for this but the libraries are already imported so..
Submitted By: Andrew Tolopko Adddate: 2008-10-02 12:40:26 inc'ing to prio 2, if only to reassess if this is still a performance problem.
Submitted By: Sean Erickson Adddate: 2009-06-02 12:45:15 another tool to look at is the Netbeans profiler. I've used it in the past with reasonable results. I found this article helpful: http://java.dzone.com/articles/jprofiler-your-java-code-could
Submitted By: Andrew Tolopko Adddate: 2009-08-05 15:01:02 With the new lib-mgmt-rework release, we just need to test performance at this point, before trying to make any optimizations. We need to test all 3 library types.
Submitted By: Andrew Tolopko Adddate: 2009-08-27 09:54:11 After profiling and redesign, memory and time performance are now deemed acceptable. See https://wiki.med.harvard.edu/ICCBL/PerformanceBenchmarks.

File Name	File Type	File Size	Posted By
No Files Were Found

Change Date	User Id	Field Name	Old Value	New Value
2009-08-27 09:54:11	Andrew Tolopko	estimatedeffort	None	Medium (<= week)
2009-08-27 09:54:11	Andrew Tolopko	active39	Active	Inactive
2009-08-27 09:54:11	Andrew Tolopko	resolution	None	Fixed/Implemented
2009-08-27 09:54:11	Andrew Tolopko	status	Open	Closed
2009-08-05 15:01:02	Andrew Tolopko	plannedrelease	None	library_mgmt
2009-08-05 15:01:02	Andrew Tolopko	active39	None	Active
2009-08-05 15:01:02	Andrew Tolopko	assignee	None	Andrew Tolopko
2009-05-27 09:06:33	Andrew Tolopko	assignee	Nobody	Removed
2009-05-27 09:06:33	Andrew Tolopko	assignee	None	Sean Erickson
2009-01-13 15:09:41	Sean Erickson	assignee	None	Nobody
2009-01-13 15:09:41	Sean Erickson	priority	3	2
2008-10-07 11:02:16	Andrew Tolopko	release	None	1.9.x
2008-10-02 12:40:17	Andrew Tolopko	_created_		library import performance

File	Date	Previous Version	New Version	Message Log	By
/branches/iccbl/library-mgmt-rework/src/edu/harvard/med/screensaver/io/libraries/DataRowParserException.java	2009-08-21 10:44:52	Diff To 3394	3395	- For [#1177], [#1581] - Refactored LibraryContentsParsers to function as one-record-at-a-time parsers, via parseNextRecord(), which replaces the all-at-once parse() method. This allows LibraryContentsLoader to handle common data persistence operations...	Andrew Tolopko
/branches/iccbl/library-mgmt-rework/src/edu/harvard/med/screensaver/io/libraries/LibraryContentsLoader.java	2009-08-21 10:44:52	Diff To 3366	3395	- For [#1177], [#1581] - Refactored LibraryContentsParsers to function as one-record-at-a-time parsers, via parseNextRecord(), which replaces the all-at-once parse() method. This allows LibraryContentsLoader to handle common data persistence operations...	Andrew Tolopko
/branches/iccbl/library-mgmt-rework/src/edu/harvard/med/screensaver/io/libraries/LibraryContentsParser.java	2009-08-21 10:44:52	Diff To 3323	3395	- For [#1177], [#1581] - Refactored LibraryContentsParsers to function as one-record-at-a-time parsers, via parseNextRecord(), which replaces the all-at-once parse() method. This allows LibraryContentsLoader to handle common data persistence operations...	Andrew Tolopko
/branches/iccbl/library-mgmt-rework/src/edu/harvard/med/screensaver/io/libraries/LibraryParser.java	2009-08-21 10:44:52	Diff To 3394	3395	- For [#1177], [#1581] - Refactored LibraryContentsParsers to function as one-record-at-a-time parsers, via parseNextRecord(), which replaces the all-at-once parse() method. This allows LibraryContentsLoader to handle common data persistence operations...	Andrew Tolopko
/branches/iccbl/library-mgmt-rework/src/edu/harvard/med/screensaver/io/libraries/ParseException.java	2009-08-21 10:44:52	Diff To 3325	3395	- For [#1177], [#1581] - Refactored LibraryContentsParsers to function as one-record-at-a-time parsers, via parseNextRecord(), which replaces the all-at-once parse() method. This allows LibraryContentsLoader to handle common data persistence operations...	Andrew Tolopko
/branches/iccbl/library-mgmt-rework/src/edu/harvard/med/screensaver/io/libraries/rnai/RNAiLibraryContentsParser.java	2009-08-21 10:44:52	Diff To 3394	3395	- For [#1177], [#1581] - Refactored LibraryContentsParsers to function as one-record-at-a-time parsers, via parseNextRecord(), which replaces the all-at-once parse() method. This allows LibraryContentsLoader to handle common data persistence operations...	Andrew Tolopko
/branches/iccbl/library-mgmt-rework/src/edu/harvard/med/screensaver/io/libraries/rnai/WorkbookLibraryContentsParser.java	2009-08-21 10:44:52		3395	- For [#1177], [#1581] - Refactored LibraryContentsParsers to function as one-record-at-a-time parsers, via parseNextRecord(), which replaces the all-at-once parse() method. This allows LibraryContentsLoader to handle common data persistence operations...	Andrew Tolopko
/branches/iccbl/library-mgmt-rework/src/edu/harvard/med/screensaver/io/libraries/smallmolecule/FileParseErrorManager.java	2009-08-21 10:44:52	Diff To 3394	3395	- For [#1177], [#1581] - Refactored LibraryContentsParsers to function as one-record-at-a-time parsers, via parseNextRecord(), which replaces the all-at-once parse() method. This allows LibraryContentsLoader to handle common data persistence operations...	Andrew Tolopko
/branches/iccbl/library-mgmt-rework/src/edu/harvard/med/screensaver/io/libraries/smallmolecule/NaturalProductsLibraryContentsParser.java	2009-08-21 10:44:52	Diff To 3394	3395	- For [#1177], [#1581] - Refactored LibraryContentsParsers to function as one-record-at-a-time parsers, via parseNextRecord(), which replaces the all-at-once parse() method. This allows LibraryContentsLoader to handle common data persistence operations...	Andrew Tolopko
/branches/iccbl/library-mgmt-rework/src/edu/harvard/med/screensaver/io/libraries/smallmolecule/SDRecordParser.java	2009-08-21 10:44:52	Diff To 3385	3395	- For [#1177], [#1581] - Refactored LibraryContentsParsers to function as one-record-at-a-time parsers, via parseNextRecord(), which replaces the all-at-once parse() method. This allows LibraryContentsLoader to handle common data persistence operations...	Andrew Tolopko
/branches/iccbl/library-mgmt-rework/src/edu/harvard/med/screensaver/io/libraries/smallmolecule/SmallMoleculeLibraryContentsParser.java	2009-08-21 10:44:52	Diff To 3383	3395	- For [#1177], [#1581] - Refactored LibraryContentsParsers to function as one-record-at-a-time parsers, via parseNextRecord(), which replaces the all-at-once parse() method. This allows LibraryContentsLoader to handle common data persistence operations...	Andrew Tolopko
/branches/iccbl/library-mgmt-rework/src/edu/harvard/med/screensaver/io/workbook2/WorkbookParseError.java	2009-08-21 10:44:52	Diff To 3272	3395	- For [#1177], [#1581] - Refactored LibraryContentsParsers to function as one-record-at-a-time parsers, via parseNextRecord(), which replaces the all-at-once parse() method. This allows LibraryContentsLoader to handle common data persistence operations...	Andrew Tolopko
/branches/iccbl/library-mgmt-rework/src/edu/harvard/med/screensaver/io/workbook2/WorkbookParseErrorsException.java	2009-08-21 10:44:52	Diff To 3394	3395	- For [#1177], [#1581] - Refactored LibraryContentsParsers to function as one-record-at-a-time parsers, via parseNextRecord(), which replaces the all-at-once parse() method. This allows LibraryContentsLoader to handle common data persistence operations...	Andrew Tolopko
/branches/iccbl/library-mgmt-rework/src/edu/harvard/med/screensaver/service/libraries/LibraryContentsLoader.java	2009-08-21 10:44:52	Diff To 3383	3395	- For [#1177], [#1581] - Refactored LibraryContentsParsers to function as one-record-at-a-time parsers, via parseNextRecord(), which replaces the all-at-once parse() method. This allows LibraryContentsLoader to handle common data persistence operations...	Andrew Tolopko
/branches/iccbl/library-mgmt-rework/test/edu/harvard/med/screensaver/db/GenericEntityDAOTest.java	2009-08-21 10:44:52	Diff To 3325	3395	- For [#1177], [#1581] - Refactored LibraryContentsParsers to function as one-record-at-a-time parsers, via parseNextRecord(), which replaces the all-at-once parse() method. This allows LibraryContentsLoader to handle common data persistence operations...	Andrew Tolopko
/branches/iccbl/library-mgmt-rework/test/edu/harvard/med/screensaver/io/libraries/clean_data_rnai.xls	2009-08-21 10:44:52	Diff To 3345	3395	- For [#1177], [#1581] - Refactored LibraryContentsParsers to function as one-record-at-a-time parsers, via parseNextRecord(), which replaces the all-at-once parse() method. This allows LibraryContentsLoader to handle common data persistence operations...	Andrew Tolopko
/branches/iccbl/library-mgmt-rework/test/edu/harvard/med/screensaver/io/libraries/clean_rnai_duplex_50440_50443.xls	2009-08-21 10:44:52	Diff To 3340	3395	- For [#1177], [#1581] - Refactored LibraryContentsParsers to function as one-record-at-a-time parsers, via parseNextRecord(), which replaces the all-at-once parse() method. This allows LibraryContentsLoader to handle common data persistence operations...	Andrew Tolopko
/branches/iccbl/library-mgmt-rework/test/edu/harvard/med/screensaver/io/libraries/clean_rnai_pool.xls	2009-08-21 10:44:52	Diff To 3340	3395	- For [#1177], [#1581] - Refactored LibraryContentsParsers to function as one-record-at-a-time parsers, via parseNextRecord(), which replaces the all-at-once parse() method. This allows LibraryContentsLoader to handle common data persistence operations...	Andrew Tolopko
/branches/iccbl/library-mgmt-rework/test/edu/harvard/med/screensaver/io/libraries/dirty_data_rnai.xls	2009-08-21 10:44:52	Diff To 3342	3395	- For [#1177], [#1581] - Refactored LibraryContentsParsers to function as one-record-at-a-time parsers, via parseNextRecord(), which replaces the all-at-once parse() method. This allows LibraryContentsLoader to handle common data persistence operations...	Andrew Tolopko
/branches/iccbl/library-mgmt-rework/test/edu/harvard/med/screensaver/io/libraries/LibraryContentsLoaderTest.java	2009-08-21 10:44:52	Diff To 3383	3395	- For [#1177], [#1581] - Refactored LibraryContentsParsers to function as one-record-at-a-time parsers, via parseNextRecord(), which replaces the all-at-once parse() method. This allows LibraryContentsLoader to handle common data persistence operations...	Andrew Tolopko

Dependent On Other Items

ID	Summary	Delete
No Dependencies Were Found

Items Dependent On This Item

ID	Summary
No Dependencies Were Found

Duplicate of items

ID	Summary	Delete
No Duplicates Were Found

Items that are duplicates of this item

ID	Summary
No Duplicates Were Found

Associated Item	Comment
No Associated Items Found
~~Add Association~~

Save changes Back

The Harvard ABCD Forge is an installation of the GForge project and is supported and maintained by the ABCD Software Developers Group. Funding is provided by the office of the Harvard CIO. Contact the forge-admin team with any questions. Our privacy policy is posted here.

MyGforge Home

Screensaver HTS LIMS