beginner questions about 3B

Bayesian analysis of blinking and bleaching

beginner questions about 3B

Postby oscar » Thu Feb 06, 2014 8:53 pm

I have just started to learn how to run 3b lately. Here are my questions.

I was following the example in the tutorial. The command is
./multispot5_headless --save_spots test_data/results.txt --log_ratios test_data/markup1.bmp test_data/test.tif

results.txt should be the output. However what does markup1.bmp do and how is it generated?

Also, I have access to a big cluster and would like to take advantage of the parallel capability. My understanding is threeB itself is not parallelized. I have to split the image and run 3b on each subarea and combine them at the end. Is there a simple way to split the image? And how do I combine the results after running multispots5_headless in "parallel"?

Thanks
oscar
 
Posts: 3
Joined: Thu Feb 06, 2014 7:28 pm

Re: beginner questions about 3B

Postby edrosten » Tue Feb 11, 2014 3:16 pm

oscar wrote:results.txt should be the output. However what does markup1.bmp do and how is it generated?


The markup1.bmp file is a binary image which indicates the area of the data to be analysed. The markup must be the same 2D size as the data. The area to be analysed is given by the non-black pixels. The image can be created in any image manipulation tool, such as ImageJ or the GIMP.

Also, I have access to a big cluster and would like to take advantage of the parallel capability. My understanding is threeB itself is not parallelized. I have to split the image and run 3b on each subarea and combine them at the end. Is there a simple way to split the image? And how do I combine the results after running multispots5_headless in "parallel"?


You are correct: 3B is not an inherently parallel algorithm. In order to run in parallel, you need to split the markup up into chunks, and analyse each chunk in parallel, then recombine them. We've recently added features to 3B to make this easier. First, you'll need one of the beta versions newer than the released version 1.1. You can get most of it here precompiled:

http://www.coxphysics.com/3b/3B_release ... 479a931ef/

However, you'll also need the program make_grid_markup. This is not currently available precompiled, so you will need to compile it from the source. You'll need to get the latest version from the git repository. The compiling instructions as the same as in the tutorial except you need to type

Code: Select all
make make_grid_markup


Once that is built, markup the entire are you wish to analyse and save as markup-all.png. Run the following line:

Code: Select all
make_grid_markup --size 15 --filter markup_small_%04i.png --image markup-all.png


This will split up the markup-all.png image to create a list of files like:
Code: Select all
markup_small_0000.png
markup_small_0001.png
markup_small_0002.png
...

each of those will contain a 15x15 square.

Note that the squares abut but do not overlap. In order to get a good 3B run, you need to expand the squares a bit and then remove the overlapping areas. The very recent versions of 3B can do this automatically. To do this, run 3B with the following extra options:

Code: Select all
multispot5_headless --save_spots results-0000.txt --log_ratios markup_small_0000.png --dilate_mask_as_filter_radius 3.2
data/image-*.tiff


this will dilate the areas to be run by 3.2 pixels, but will remember the original square, so during reconstruction, it can be used for filtering purposes. Run 3B on all the small markup images for an appropriate time.

You will now have a number of output files such as results-0000.txt. These can simply be aggregated into a single large file from the terminal:

Code: Select all
cat results-0???.txt > results-all.txt


Now, load the aggregated results file using the plugin. If you check the "filter" box in the load dialog, then for each run, it will filter out anything outside the original square. This will cleanly combine the data.
edrosten
 
Posts: 42
Joined: Wed Jul 18, 2012 2:54 pm

Re: beginner questions about 3B

Postby oscar » Mon Feb 17, 2014 6:51 pm

Thanks, Ed, for the detailed instructions. I will give it a try. I have gimp installed. How do exactly I generate the markup file though? I can't seem to find the option from the menus.
oscar
 
Posts: 3
Joined: Thu Feb 06, 2014 7:28 pm

Re: beginner questions about 3B

Postby edrosten » Thu Feb 20, 2014 10:28 pm

Take a single frame or a z project of the data (this example is taken from the test data):

AVG_test_data.png
AVG_test_data.png (4.28 KiB) Viewed 28788 times


Load it into the GIMP:

image-1-basic.png
image-1-basic.png (22.78 KiB) Viewed 28788 times


Bring up the layer dialog and add a transparent layer in front of the image:

image-3-layer.png
image-3-layer.png (38.62 KiB) Viewed 28788 times


Set the colour to white, and select the pencil tool (not the brush) and set it to an appropriate size. Draw over the areas you wish to be analysed:

image-4-dot.png
image-4-dot.png (62.99 KiB) Viewed 28788 times


Now, select the back layer (the original image) and fill it with black:

image-5-fill.png
image-5-fill.png (30.65 KiB) Viewed 28788 times


Save the resulting file as a PNG by going to File>Export:

AVG_test_data2.png
AVG_test_data2.png (283 Bytes) Viewed 28788 times


If you use the final image as the markup, then the area under the white dot will be analysed.


This is generally the method I've used for creating markup.
edrosten
 
Posts: 42
Joined: Wed Jul 18, 2012 2:54 pm

Re: beginner questions about 3B

Postby oscar » Mon Mar 17, 2014 9:15 pm

Hi Ed,
I followed your instruction to create markup files and got a better understanding of the process. Still have a few questions:

1. So the idea is to divide up the whole image into subareas. This way 3B analysis of subareas can be run at the same time in parallel. To do that, the smaller jobs still have to be initiated manually, Am I correct?

2. If there are a lot of tiff files to be processed, with the command "multispot5_headless --save_spots results-0000.txt --log_ratios markup_small_0000.png --dilate_mask_as_filter_radius 3.2
data/image-*.tiff", how are the files processed? Are the subareas of each frame being processed simultaneously? The next tiff image will be processed once all the subareas of first image is done processing? So if there are 100 60x60 images with markup size of 15x15, the program will generate 16x100=1600 result files?

3. My understanding is that, the image quality will be improved (signal noise ratio enhanced) after running 3B which will result in a new set of images with same dimensions as the inputs. Is that right?
oscar
 
Posts: 3
Joined: Thu Feb 06, 2014 7:28 pm

Re: beginner questions about 3B

Postby edrosten » Mon Mar 24, 2014 9:30 pm

oscar wrote:Hi Ed,
I followed your instruction to create markup files and got a better understanding of the process. Still have a few questions:

1. So the idea is to divide up the whole image into subareas. This way 3B analysis of subareas can be run at the same time in parallel. To do that, the smaller jobs still have to be initiated manually, Am I correct?


That's correct. It can be automated, but the precise mechanism will depend on if you're using a cluster and what clustering software you're using.


2. If there are a lot of tiff files to be processed, with the command "multispot5_headless --save_spots results-0000.txt --log_ratios markup_small_0000.png --dilate_mask_as_filter_radius 3.2
data/image-*.tiff", how are the files processed? Are the subareas of each frame being processed simultaneously? The next tiff image will be processed once all the subareas of first image is done processing? So if there are 100 60x60 images with markup size of 15x15, the program will generate 16x100=1600 result files?


Once you've used the make_grid_markup program, each file like markup_small_0000.png, make_markup_small_0001.png etc will be treated separately. This means if it gets split up into 50 separate images, you will need to issue 50 separate commands, and that will generate 50 separate results files. The program won't automatically move on when it's finished, it will simply stop. Therefore, you need to generate those 50 commands to run.

In general, knowing when to stop MCMC is an open research problem. What we generally do is to let each individual run go for 4 hours. At the end of the run, we automatically reschedule a continuation run with a slightly lower priority. In practice this means each area gets 4 hours of CPU time, then when all areas have had the 4 hours, it will add on another 4 hours etc. We let it keep going until it's reached enough iterations. This ensures fair queueing on the cluster. The precise details of this will vary depending on the system you're using.


3. My understanding is that, the image quality will be improved (signal noise ratio enhanced) after running 3B which will result in a new set of images with same dimensions as the inputs. Is that right?


Not quite:3B takes in an image stack (200 images or so), and in the marked area it computes a super resolution reconstruction using that stack. The output is a list of (x,y) pairs indicating possible fluorophore locations. This list must then be turned into an image for viewing. The ImageJ 3B plugin can do this. Typically the image created from these (x,y) pairs is just big enough to hold all the data, however, you can arrange it so that it covers the entire image area easily enough. The output is typically considerably larger, however. Input images are typically in the range 100-200nm/pixels, but an output image will generally be generated at 10nm per pixel to give a smooth reconstruction.
edrosten
 
Posts: 42
Joined: Wed Jul 18, 2012 2:54 pm

Re: beginner questions about 3B

Postby hibaaryan » Tue Jan 06, 2015 4:47 pm

We're comparing the results imagej plugin and the results of the compiled program, we're using the same configurations on the mustilspot5.cfg but the results aren't the same, the later reconstruction seems to have less quality. The version of 3B is 1.1 in the two cases. Could you please advise us about it?

Another question, I'm just wondering if the 'iterations' shown the result file of imagej plugin are the same as the 'iterations' that are shown in the result file of the compiled program, i.e. does the algorithm needs to be stopped by the "Iteration: 50(200)"?
hibaaryan
 
Posts: 1
Joined: Tue Jan 06, 2015 4:34 pm

Re: beginner questions about 3B

Postby hoha » Fri Jan 09, 2015 3:10 am

Hi hibaaryan,

I did this questions some time ago and ed and susan had replied http://superresolved.com/viewtopic.php?f=10&t=30222, I didn't understand what are u looking for?

Rewards,
Haydee.
hoha
 
Posts: 9
Joined: Mon Apr 14, 2014 9:24 pm


Return to 3B

Who is online

Users browsing this forum: No registered users and 1 guest

cron