[Recording] This conference will now be recorded.
 
[Emily Lemagie] Awesome. Good morning everybody and welcome to another EcoFOCI Seminar Series. I am Emily Lemagie. I'm co-lead of the seminar series with Deana Crouser. 
This seminar is part of NOAA's EcoFOCI bi-annual seminar series, focused on ecosystems of the North Pacific Ocean, Bering Sea, and U.S Arctic to improve understanding of ecosystem dynamics and application of that understanding to the management of living marine resources.
Since 1986 this seminar has provided an opportunity to discuss and promote conversation on subjects pertaining to physical and fisheries oceanography and regional issues in Alaska marine ecosystems. You can visit the EcoFOCI Webpage for more information. And we sincerely thank you for joining us today as we're transitioning for a hybrid online and in-person seminar. So thanks to everyone both here in the room and in home or in your offices. You can look for our speaker lineup on the OneNOAA Seminar Series or on the NOAA PMEL Calendar of Events.
And if you missed a seminar, you can catch up on PMEL's YouTube page. It takes a few weeks to get them posted but the seminars will be posted. I ask that you please make sure that your microphones are muted and your videos stay off, throughout the talks. I'm excited to introduce our two speakers. 
 
We have Rob Suryan and Dave Kimmel. And these presentations will cover ecosystem-based management in the Alaska Fisheries Science Center, from how ecosystem data is being collected and how it is being used. And our first speaker is Robert Suryan, he's the Program Manager for the Alaska Fisheries Science Center. And specializes in integrated ecosystem studies to understand population and community dynamics in response to changing food availability and ocean climate. And then after his talk, we'll have time for short questions before we transition to our second speaker.
 
[Emily Lemagie] Go ahead Rob.
[Rob Suryan] Great thanks Emily and thanks Deana for organizing this and I appreciate the invitation. So I'm going to focus on the part of EcoFOCI and part of the ecosystem studies, at Alaska Fisheries Science Center.
Of course there's lots of people contributing to ecosystem studies overall, but for today I'm going to focus on just the recruitment process alliance. And kind of what the purpose of it and what we do within the RPA. So the Recruitment Processes Alliance leads are: Libby Logerwell, who's acting at the moment and then Julie Keister from University of Washington will be the new hire into that position starting soon. And we're looking forward to that. Ed Farley is also a lead, along with Phyllis Stabeno and myself.
 
 
Okay and so we're starting these presentations off with our ABT. So here's mine, changing climate affects fish populations and fisheries management needs to respond. But knowing how to respond is challenging. Therefore we study how climate ecosystem processes affect recruitment of juvenile fish which are a primary driver of population change.
 
Okay so what is the RPA? It's the, it's four AFSC programs, large vessel surveys that are offshore. So the primary programs for this effort are the recruitment processes, FOCI in Seattle, Pacific Marine Environmental Lab, in Seattle, the Recruitment, Energetics and Coastal Assessment here in Juneau. And then Ecosystem

Monitoring and Assessment, also in Juneau.
And these cruises are different from other cruises that AFSC conducts because we collect a wide range of data on all ecosystem components from physics to whales. And so this includes physical oceanography, water chemistry, phytoplankton, zooplankton. We focus primarily on juvenile fish and crab, so the other surveys such as the bottom trawl surveys and others are focusing on the adult life stages. We're focusing primarily on the juveniles. At least for groundfish, but there are exceptions, forage fishes all life stages and then also marine birds and mammals. But wait there's more to this whole effort because we have a lot of collaborators that we work with.
 
So part of the RPA is also these collaborations and some of our programs also focus on coastal and nearshore studies. And these very strong partnerships within the AFSC and outside of AFSC. So we have as far as collaborations throughout AFSC pretty much every program where we have some type of collaboration with, but it especially we work closely with the Fish Behavioral Ecology program in Newport. And also this Shellfish Assessment Program in Kodiak. The RPA has been a key player and a lead to each of the integrated ecosystem research programs that have occurred so far in Alaska. Including the Bering Sea and the Gulf of Alaska, and those arctic integrated ecosystem studies. Those are primarily funded by North Pacific Research Board but also collaborative funding with NSF and also collaborative research with NSF and BOEM. 
 
And then we also are the lead for the Gulf Watch Alaska Long-term Ecosystem Research and Monitoring Program. So as opposed to those previous programs I mentioned those are generally in the five-year time frame for those integrated ecosystem research programs. The Gulf Watch Alaska has been ongoing for 10 years now, including though a lot of data sets that were collected actually as part of the GOA integrated ecosystem research program and prior oil spill related studies. 
So some of those time series that are dated back to decades. And actually GAK1 is now 50 years, a little over 50 years old. But the RPA leads this effort and we just got funding for another five years. So we'll be going to a full 15 years, we're hoping to make it to 20, but that's uncertain at the moment but it's led by people within the RPA but it's the primary investigators involved in these studies are non-government and non-NOAA government collaborators. There's actually relatively few NOAA people but we actually use a lot of the data coming from Gulf Watch Alaska for, for purposes related to the RPA and NOAA s mission.
 
Okay so along with those specific studies that I mentioned there's a, those have their own set of surveys and cruises but the RPA manages a suite of annual or biennial cruises. And those include for the Arctic, the Distributed Biological Observatory. And then the those maps shown here are the surveys that were conducted for 2022 in the Bering Sea and Gulf of Alaska that were RPA led.
 
 
In the Bering Sea there's the Northern Bering Sea Survey which is annual along with the DBO Survey. The Southeast Bering Sea is every other year during even years, and then there's the 70 meter isobath which is both mooring maintenance and replacement, and then also sampling. For along the 70, 70 meter isobath in the Bering Sea. And that's two cruises per year but they're pretty short. So spring and fall. And then in the Gulf of Alaska there's the Western Gulf Alaska Survey, in the odd years. And Southeast Coastal Monitoring which is annual in southeast Alaska.
 
 
And then some of those other...[voice interrupts] Sorry, was there a question?
 
 
Oh okay.
And then some of the other surveys that are shown there on the map that occurred this year, which is an off year for the Gulf of Alaska for RPA surveys, are the surveys conducted by the Gulf Watch Alaska, which are annual. And then also the nearshore beach sane surveys for juvenile fish, primarily Pacific Cod and Pollock. And then also the Southeast Coastal Monitoring, which is an annual survey.
 
 
So the RPA vision and mission, overall the vision of the RPA is to advance the mission of NOAA Fisheries through ecosystem monitoring and process studies that improve understanding and forecasting of ecosystem and fisheries dynamics. And then we apply that understanding to management of living marine resources and all of Alaska's LMEs. And then our, the mission how we accomplish that vision is to use observations from seasonal and biennial fisheries-oceanographic research and monitoring surveys, laboratory experiments and modeling to determine the impact of climate on ecosystem dynamics and fisheries outcomes.
 
So we kind of look at the RPA, kind of operating as the pyramid here where we have at the base of the pyramid really are these long-term moored observations, cruise surveys and time series, that we cultivate. Process oriented research on top of that, advanced technologies for novel analytical approaches on top of that, to develop mechanistic understanding and provide advice.
 
And that top of that pyramid, that advice, it funnels into all the RPA activities that go into actually a wide variety of ecosystem management but in terms of ecosystems based fisheries management is through the council process. And these go into the, more ecosystem large marine ecosystem based assessments, or reports of the ecosystem status reports, for all of the LMEs and then also they go into single species based stock assessments too.
 
 
In terms of both indirect use of/in helping to inform outcomes of specific models but also some direct use being applied where they're quantitatively used within a model too. And both have particular value depending on your interest and questions. So these are part of those include risk tables, but also the ecosystem and socio-economic profiles.
 
 
So now I'm just going to mention a few case studies that are related to that type of effort. And this is a situation where, there's again lots of data that are collected on the ecosystem and the ocean conditions that many of those might be indirect assessments of - is it a warm year or cold year? etc  
But these are some examples of where we take a variety of those data inputs and use them directly to try to get, for a species of interest. For/from a management perspective so a direct application and direct measurements of this species of interest. So here's a good, an excellent example of Pacific Cod spawning habitat. So this uses data from the Gulf of Alaska 1 mooring, just outside of Resurrection Bay.
And lots of analysis to show how that relates to water temperatures throughout the gulf. And how broadly applicable using that time series is and also experimentally derived temperature range for hatching success a Pacific Cod within, so laboratory experiments to develop a suitable spawning habitat projection, for a given year.
 
So here's an example on the right that shows that the color codes of hatching success probability; up to 0, from 0 to 0.5, depth along the y-axis, and then month along the x-axis. So it gives you where you see yellow, you see, better spawning probability, better or hatching success probability. So as far as the Pacific Cod spawning habitat and you can see in 2015 and '16 there was a lot of really poor spawning and hatch success probability for Pacific Cod. But in recent years promising that there's some improvement which is good to see for 2021 and 2022.
 
Here's another example where/of direct use. And this is what two of our surveys; the Southeast Coastal Monitoring and the Northern Bering Sea Survey, are strong partnerships. And couldn't be completed without the partnership with the Alaska Department of Fish and Game. So some of that, the focus, it's a you know, a standard RPA ecosystem study or a survey but it a lot of this focuses also on salmon for the state of Alaska. And so here's an example of in Southeast Coastal Monitoring, where a catch per unit effort and temperature is used to/for pink salmon forecast models.
And you can see the relationship on the left of the model and then the heart, the forecasts on the right and the industry and salmon fisheries managers, are very much interested in getting this annual forecast, for the following year. So you can see the one for 2023 and you can see how well that forecast does in most years. And then on the right is an example of this, with the graph shown are for Chinook, but it's a both separately conducted Chinook and Chum salmon forecasts for the Yukon River.
And this is using catch per unit effort or estimates of juvenile abundance and then projections for future returns for Chinook salmon in the Yukon River. And you can see there's a fairly, a very good relationship and good projections from those models.

Here's another example of contribution that's related
that the recruitment processes alliance helps to facilitate, but is primarily conducted by our collaborators with Gulf Watch Alaska and the U.S Geological Survey. So this is some work that Yumi Arimitsu and Scott Hatch and others where they're collecting, um, juvenile Sablefish from the diet or from seabirds. So this is here's Rhinoceros auklets and showing foraging trips for auklets and kittiwakes from Middleton Island.
 
 
So they're out sampling fish and throughout this region in the North Eastern Gulf of Alaska. And from collections by the birds brought back to the colony and then preserved for analysis, you can see the relative growth or size at a particular date for the summertime of juvenile Sablefish. So here's a plot now that Yumi has made for the length anomaly, for estimated size on July 24th and you can see it varies considerably between warm and cold years and a good example of, what how different this can be is in 2020 and 2021, shown in the picture here, the very dramatically different sizes of the two fish. So this isn't a, this information is being used as part of the Sablefish ESP, but it/and it's still under development and that's the case for a lot of these. They're constantly being refined and developed.
 
And then here's an example of juvenile Pollock energy density, versus recruitment. So this is some work that was initiated by Ron Heintz and Eubet Siddon, with the, in the Bering Sea. Showing again, trying to forecast based on energy content of juvenile fish what age-1 recruitment is going to look like in the future.
 
 
And it's interesting because when this was initially analyzed in one of the earlier publications there was a pretty strong relationship, with just out considering warm and cold years, but as they collected more data you'd see there's kind of different relationships between warm and cold years. And then collecting more data there's outliers. And in some respects the kind of frustrating that you have outliers and your relationship might not be quite as strong but, in other respects those outliers are super interesting from a process perspective. 
 
So there's a lot that we can learn from those, from the outliers and also it's highlighted in the fact that these relationships are not stationary. We would expect them to change over time. And especially given that the way our/the climate is changing in Alaska, as shown here in terms of September ice coverage and Chukchi Sea, the occurrence of heat waves and the increased frequency in the Gulf of Alaska, there's no reason to believe these relationships should be stationary and we need to continue to evaluate the work and these that we're doing and continuing these long-term monitoring efforts
and and process focus research. So as far as our overall focus and direction, just in summary, the RPA and all of the ecosystem programs at the Alaska Fisheries Science Center, provide the why. Why are populations increasing or declining? Why are fish in poor condition? And for example, why are harmful algal blooms occurring?
 
 
Again we're constantly evaluating, re-evaluating the processes because those allow us to mechanistic understandings to actually make projections and understand what the future might look like, in the near-term and the long- term. But we're also adaptive too, so constantly adding new sampling platforms one of which Dave's going to talk about. And to provide data, more, in a more timely manner and as soon as possible and automate some of these processes. 
 
But also adding gear too, so for example a small mesh beam trawl has been added to the surveys in the Bering Sea this year. The book the Southeast Bering Sea and the Northern Bering Sea and those/those actually cruises in particular have an enormous demand for samples. They're very highly sought after for lots of researchers within AFSC, and outside of AFSC and it's become quite challenging to coordinate all the data requests for the end, for those cruises. And then also the goal here is to contribute to both, you know, tell/describe what has happened. But also provide some early warnings and also forecasting like some of those examples I showed.
 
And just to end with a huge thank you because this is a massive effort from many people. AFSC and PMEL leadership for supporting this, all the PIs that are involved in developing, designing studies, analyzing data, the science and vessel crews and laboratory staff which are too numerous to list but is critical to this effort. And then of course that the cruises are no simple task either, but the logistics and administrative staff to make those happen too.
All right, that's it, thank you.
 
 
[Applause]

 
[Emily Lamagie] Thank you. We have time for questions for Rob. So if there's anyone online, you can enter them into the chat and do we have any questions from the room?
 
[Silence]

Rob, thanks for sharing the, the different examples and do you have any comments on specific priorities or projects coming down the pipeline, that you're most excited about?
 
[Rob Suryan] Thanks Emily. Yeah I guess there's quite a few projects that are kind of under development and kind of exploratory. And I think there's a couple that I think are particularly interesting and exciting. 
 
And I think and some of that is for, like I mentioned with the work with the small mesh beam trawl and the kind of pelagic benthic coupling work in the Northern Bering Sea and Southeast Being Sea.
That transition, is one of the priority areas that we think is important for study. And so I think some of the work coming out of that, is particularly interesting. I think some of the work that you're going to see next from Dave is super exciting in terms of there's like, as I mentioned the effort to make these data as, to provide data as rapidly as possible,
as soon as the survey is completed.

And that has been a, we've been pretty successful in that, but it's there's, also a lot of work to be done there. So yeah I would say what Dave's going to present is super exciting and kind of shows where we're going with some of this effort. There is also a question about the juvenile energy density. So there's a couple different ways that, we actually, about three different ways that we're good at that. 
So one way and the example I showed was from bomb calorimeter, calorimetry so just kilojoules per gram, in wet mass but we can also convert to dry mass basis. We also use a
Sulfo-Phospho-Vanillin SPV Method

which is a calorimic.

And it looks at, that's a more rapid assessment and much cheaper to do because it is a, it's simple and faster and that's another one of those methods that we can produce results, within a week after a cruise versus, sometimes months. And then there's also more of the proximate composition looking at different lipids, within the, within the organism.
 
And that's even more involved but also, that's that more detailed lipid information, energy information, provides answers to specific questions. So we use a combination of all those between working with the lab here at, in Juno, but also with Louise Copeman in the Newport lab.

[Deana Crouser] Awesome, thank you so much.
[Rob Suryan] Okay, I'll stop sharing.

[Emily Lamagie] All right, our next speaker is Dave Kimmel. Dave is a Lead Research Oceanographer at the Alaska Fisheries Science Center. His area of expertise is biological oceanography, zooplankton ecology, coastal ecology, climate
impacts on ecosystems and qualitative ecology. 
[Indistinct]

[Dave Kimmel] Greetings virtual audience.
Okay this is the pointer. Does this 
work on the screen?

[Deana Crouser] Mm-hmm. It does!

[Dave Kimmel] Thank you to somebody. Ok then.
Greetings everyone. It's a pleasure to be here and talk to you. I'm going to essentially give you the talk I gave to PICES meeting recently.
 
 
And what I'm going to talk about is how to automate a rapid zooplankton assessment for use in ecosystem-based fisheries management. And I want to acknowledge my co-authors here, Deana Crouser, who works as part of the zooplankton team has been working side by side with me to get this thing off the ground. And then our colleagues in Poland that have been generating a very large annotated image data set for us. And then my colleague Hongsheng Bi, has been working to help us with the AI portion of this, that you're going to see today. So I just want to acknowledge those folks, and here's my ABT
which is that zooplankton are very important critical ecosystem components because they connect our primary producers, phytoplankton to higher tropic levels. In other words they're the gateway between all of the energy that's fixed in the bottom of the food web. And there are important indicators of the

ecosystem status that are used in management. But the problem is that traditional sampling requires significant time and expertise to generate these data. Therefore we need new methods of assessment that are required. Or new methods of assessment are required to ensure managers receive timely information on zooplankton or I'm out of a job. So what I want to talk to you about today is the zooplankton information to fisheries management and why that's a problem.
 
Ok. Rapid Zooplankton Assessment is what was the solution that was designed before I sort of joined EcoFOCI. And we've been working to assess this, figure out if it works and how it can be used in management and then, figure out if we can automate this approach which is our work in progress. And then some future work and some parting thoughts on sort of where this, this is headed. 
 
So fisheries management in Alaska as Rob sort of alluded too, is a big deal. There's a lot of jobs in the seafood industry, there's a lot of money at stake and it exists on an annual cycle. So the annual cycle is sort of Rob gave a beautiful introduction. It's all of this information is coming in, from ecosystem models, survey data, all these different information is then being funneled down into this Plan Team.
There's a review and at the end we get this management outcome which is the total allowable catch for the Fisheries. And so what you have to understand is it's a huge funnel, there's a lot of information going in there, but the key thing to remember here is it's an annual cycle. Which is going to be a problem for us zooplankton folks, as you'll see in a second. That's because we have a very large footprint. We're talking about very large marine ecosystems that we're responsible for. 
 
That's five, that requires a lot of vessels, a lot of sampling as Rob told you. We have Institute data, survey data, laboratory data, time series, maps, energetics, all this stuff has to go into these reports. And it has to be done on that annual cycle. Otherwise it's old news for next year. And so if you're a zooplankton person, that's bad because you have a lot of samples but you need time to process those samples. You need experts to count them, to go through them, identify the species and that takes a lot of time.
 
So the data were not being applied to fishery management during the annual cycle. And that became a problem because we're, you know we know this information is really important, but we have a problem you know. We collecting, sometimes we get 1500 jars and we send them to our colleagues in Poland and they sort and identify them and the data comes back and we check it and make sure it's okay. And then we put it into our database and we start to do some analyses and it's next year.  
Then what? So the solution was or, or the problem was collecting the zooplankton data are easy. Turning it into a meaningful data is quite difficult. So it requires significant expertise in time. And unfortunately taxonomy is not exactly a growing field. Which is personally a tragedy I believe, but that's one man's opinion. But the solution we came up with, is why don't we try a rapid sort on board the ship. Approximate the standing stock of the important groups that we know relate directly to fish, that they're feeding upon, and we provide that information in the form of a zooplankton count.
 
So the way we do that is, we collect a regular sample.

Where is it?

Ah, there we go!

So we collect a regular net sample and we filter that down into a jar. One of these jars give us a poll, and we eventually count it. The other one aboard the ship we take some sub-samples and we start to count the large copepods and you can see them here at Calanus, Neocalanus. We count the small guys you know, the little fish got to eat too. Pseudocalanus, Acartia, Oithona, and then Euphausiids and this information is then put into these large course categories. And we can summarize that, for I'm sorry, go back a bit.
We can summarize that information and then what we can do is we can compare it to the actual counts that we do. And so I did that using, and I'm in the middle of this analysis, but I did a Bayesian analysis, it was a simple analysis, to x.

If I take the RZA can I predict the actual eventual abundance and can I do that effectively? And the answer is, why fit a bunch of different models, and the base model is basically just, can the RZA predict the predicted final count?
The answer is yeah, it sure can. And that is this base model here, down here and you can see the R-square is about .6 - pretty good for ecological data. Then I asked the question, well, does it vary by large marine ecosystem? Does it vary when you add in a different season or does it vary by whom student, who is doing the sorting? And the answer is that basically the best model is when you account for the sorter. And that's because, believe it or not, it is expertise, when you go and count these things. So however, even if you account for that and you recognize that some people are better at it than others. 
In general, this works really well. Okay so it matches our counts and I'm even more, bullish about this because my colleagues in Canada on DFO, did this on their cruises this year and they're producing the same results. Okay so this is a viable tool, that we can use, it produces information that matches what our eventual counts get. And it's usable for ecosystem management. And I will use it for ecosystem management. Well each year we produce a map and we say okay here's the distribution of the large, the small, and the Euphausiids. And then we can put that into our time series over here.

I don't know what I'm doing. All right anyway. We can do it over here and you can see that for the black dots which are the counted samples that eventually come from Poland, and the blue triangles which are the RZA that these match up pretty good, you have some Euphausiids problems here and there because, we tend to get boom and bust with Euphausiids when it comes to whether we get them in the net. But for the most part these data match up very well, so we think first and foremost we have a viable tool. But the second question is why in the world do we want to automate this?

And I like to call upon legends and this is the legend of John Henry who is a steel driving man and his job was to pound in these railroad spikes. And he was really good at it, but eventually along came the steam engine. And that led to a very big contest, historical contest, swearing, and John Henry 
prevailed. But it cost him his life. I don't mean to be so dramatic and to say that we're going to take all the taxonomists out one day and not use them anymore. Just to say that eventually automation comes for us all. And so what we want to do is remove that sorter impact, on the RZA. So in other words we we do train individuals to go out, but some are better than others and we know that affects the results a little bit.

We want to reduce the time, now we do the RZA about every other station, it takes time to do on board, but if we could just take an image, scan that image count the individuals, identify them, we could do all the stations. Okay we can get much more information. Also we want to provide an easy to use tool to produce RZA and the other large marine ecosystems. Unfortunately zooplankton collection is, it's in peril across a lot of these other fisheries science centers because it takes a lot of time to collect/process the data. And therefore I want to sort of rescue this as a tool to help other fishery science centers.

And then eventually we want to take what Rob is talking about and provide additional information on size, lipid content for additional context. So we can add more information by analyzing these images. So there are a lot of reasons to automate the RZA. So how are we going to automate the RZA? Well we want to turn to artificial intelligence and machine learning and what we want to do is take a large annotated library. We want to annotate those images and we want to train an algorithm to identify this. So we have a great resource in our colleagues in Poland. So they've kindly set up a microscope in place and when they're done counting and identifying the samples for us they image them. And by imaging them then we can use this commercially available free software called LabelMe. 
This is extremely tedious but it's the way to get your algorithm going, you draw some polygons around your little animals and you do this repeatedly. And we produce lots of nice images like this and then after we're done we have a large annotated library. Because of the level of tediousness with this, Poland is an amazing partner to do this. Because they're identifying the individuals and usually it's just one or two individuals in the lab trying to do this. We have a whole lab that's working with us so it's a really nice partnership and they've done an amazing job. The next thing is we do is we develop an algorithm flow.
And our algorithm has two particular pathways in it. The first is it detects the background. And this is a sort of small lightweight neural network. And it basically goes 'is it a dark background or a light background?' The reason we do this is that some zooplankton are easier to see on a dark background or some are easier to see on the light background. So first it does is detect the background says light or dark, that's easy.

The next is we need a convoluted, convolutional neural network, it's a region-based thing. And it has two steps. First it says I need to detect the thing that I'm pointing at the screen. And I know people at home can t see what I'm pointing at, so sorry. I'm sorry. We are doing a regional proposal network and that is to identify the actual targets. Yes please help. Thanks.
Thank you.

Ah!

There we go. Thanks! 
Hold it down, gotcha.
Okay thank you. Ah, so region of interest is ROI detection and that's our Regional Proposal Network. And that basically says what's our target in the image. It's not as easy as you, it looks easy for the human you can look at it and go yeah that's copepod. Not so easy for the computer. So we have to train the algorithm to do this. And then once it's trained it will orient these guys and then it will do a classification based on the residual neural network that we train on all of those images that we've done in Poland. Okay and so what we're doing now is we're essentially training up this algorithm and we're testing it. And so I'll show you some results from that. Now it's not advancing.
 
And I'm doing this big technology project. [laughs ]
Okay here we go! So here's the results for scene classification and as you can see the algorithm is very good at determining whether it's a light or dark background. Which is you know it's a small step, but it's very good at it. Sometimes when it's, it thinks it's a dark background when it's a little bit light so there's a little bit in there that we need to train but once we got the algorithm going, the success rate was very very very high. Not the same for classification. It's much more difficult to classify individuals.

So here's our initial training. We were getting after about, you know 20,000 iterations and accuracy rate of about 85% or so, in our course categories. And when we increase the iterations which requires more GPU computing time and more data. We're now getting up into the upper 90s with our course categorization. So it's going pretty well and this is just for the RZA category: small copepods, large copepods, Euphausiids, keeping mass
a couple other things.

So it's not a very complex algorithm but so far
it's functioning. So what I'm going to show you

now is how the algorithm looks when it's working,

so I'm going to play you a little video and the video is going to...Ok, what do I do? Go over here? I have to go over to, oh there's a little...
guess you can't click on that.

Ah, right there! So now it's going and what
 you'll see is the algorithm is going to look

at this image and then it's going to mask
 these individuals and it's going to count them.

In some cases it's going to look really good.

And in some cases it's going to look massive. Okay, playing, there we go, all right. So let's keep an eye on it as we're watching it and you'll see a mask and count the individuals. And it'll get a little spit out at the bottom, it'll tell you the number of individuals that are in there and what category they belong to. That one was pretty good.

Now we'll switch over to a light one, it has a little more problem with the light one. You see all the different bits and pieces in there, see not so good. And then it actually does a little better depending on how many individuals are in it, depending on how many different types of species you can see I'm using, single species images here. When we mix them up it does a little worse. But this is basically how the algorithm works. And this is designed as a standalone application that we can take to sea with us and process the images while we're at sea using a small GPU. These are Sea Butterflies and you can see that they're very easy to detect. These are Chaetognatha. You can see that they're also fairly easy to detect.
 
At least partially. And so the algorithm is working. I'll show you some results on classification and counting. But overall, we feel like we have a viable start to our activities here. This is final one and looking pretty decent.

All right, so. How does it work? Well answer is, yeah okay. In our first iteration it is pretty good at detecting large copepods, as you might imagine and it's really good at detecting your own, really good at detecting large copepods. Not so good at small copepods. It gets a little confused with that size difference, so we have to hone that in. It's pretty good at detecting Euphausiids, and very good at detecting Chaetognatha but it's not so good at counting. Which you'll see over on the right side, that's a bit of a scatterplot but the positive thing is that we are moving in the right direction. In other words that line is going upwards to the right, which is what we want, it's not flat or down.

And so the algorithm can be improved, to get these counts a little better. And our intention was to have Deana take a bunch of images from this fall cruise, on the, on the Dyson. And she got a grand total of four samples before the Dyson crapped out, and had to go back to port. So we will be back at this next year and generating images on board the ship and testing mixed images, to increase the algorithm. Which is something we'll try over the winter. So, to conclude, based on our beginning set of training data we have a viable classifier for the RZA Category. So in other words we can generate RZA data from an image. Okay humans right now are better at it, but in general the algorithm is pretty good at classifying and being able to detect the categories that we want. 
But it's going to require more data. Okay, the answer to all these AI questions is more data. So annotate and train. Annotate and train. So last time I talked to Hongsheng which is about a month ago, he said the improvements are now getting better but they're more slight because the original, the classifier itself works, it has to get better with more information. We need to develop a better streamlined index generation at sea. It's not so much fun to try to take an image of a sloshing dish of zooplankton, when you're sitting in a microscope at sea.
So we need to think of some ways to do better at that. We thought about a flatbed scanner but also not so great at sea, with moving around. So I posed this question to folks at the PICES, there wasn't a lot of ideas, so I'll pose it again here. And I'm open to discussion. We are using a microscope but that does affect our depth of field, as well as well as our magnification. So we need to sort of come up with a couple ways to do that. How robust is the algorithm?
That's a real key question because are we over tuning it to the samples we get from our system? Can it be applied to other systems? We plan to add in the field generated images of course, that's the ultimate test. But is that going to destroy our algorithm? Will it require more training, adjustments for a new algorithm? 
 
I pain myself to think about whether that will be the case but that is the way this works and so essentially we will keep throwing data at this and hopefully in the coming year have a real, real world test of whether or not the RZA can be done through image analysis only.

Which for those of you that do the RZA at sea, I can hopefully hear you saying yay so that you don't have to sit in front of the microscope and count all of these small copepods, in order to generate this data. And many of you listening online and in the room here have done RZAs over the years, so we thank you very very much. And I'm done! 

And this is my cruising with Covid picture, from 2021, so I think yes.

Thank you!

[Applause]

[Emily Lamagie] Awesome! Thank you, Dave.

Oh, we have a question from online.
Why not use an automated imaging system designed for zooplankton?

[Dave Kimmel] Such as?

The ZooScan I'm assuming you're talking about?
[Emily Lamagie] Yeah, someone was asking if

you'd considered using the ZooScan?
[Dave Kimmel] Yeah. So, the ZooScan is really just a scanner. So we could use that to image the zooplankton, but the algorithm itself is basically developed much in the same way. So we chose to do our own algorithm and work on that so we didn't have to spend all the money to buy the ZooScan is essentially

the reason. But the ZooScan, I had one many years ago and this was before AI was kind of a huge thing and it worked okay, but it required a very large amount of human input to generate the images that were usable by the ZooScan. Maybe the technology has changed. I want this to be take an image, as easy as possible, run the algorithm make it as robust as possible so that as little human intervention is needed.
If someone has to sit in front of the scanner with the cactus hair make sure that all the individuals aren't touching and make sure that all the resolution is the right way, it defeats the purpose of what I'm trying to do. So.

[Emily Lamagie] Are you familiar with VPR?

[Dave Kimmel] Yup, yeah. We have a used the VPR, we actually work with some folks that are using the VPR.
We have an in situ camera that we purchased called the Cpix, that we're going to be using, and have used. We also have a plankton imager that we work with Hongsheng on, called the Plankton Scope that we plan on taking out.

So there is, a I didn't talk about it here, but there's a parallel in situ imaging pathway that we're working on, as well. We have a ton of images we actually our new program manager, Julie Keister, 
has been working on this as well. She has a camera that's been working out in in, Hood Canal.

Is that right, Deana?
Hood Canal, to look at differences of zooplankton in oxygen zones. So yeah. We're familiar with all of that stuff, yep.

[Emily Lamagie] Another question, from Elizabeth Billows.

I'm curious if the vessel has fisheries acoustics and if so, have you looked into using that data to compare, develop, compliment or supplement your Zooplankton abundance and distribution estimates?
[Dave Kimmel] Yeah, we have. We have an acoustician on staff named Adam Spirit who's been looking into this for us. We're actually quite interested in new euphausiids, so this summer we went out and did some thought trawling, and it is to look at the relationship between acoustic sign and euphausiid abundance. And then we're hoping as AI begins to develop a bit more for the acoustic streams, that we'll be able to take the acoustic data from the Oscar Dyson and looking for zooplankton signals, as well, 
to complement this. So, good question.
[Emily Lamagie] Are there any other questions from the room?
[Participant] Hey. 

[Dave Kimmel] Yep.
[Participant] From just the technical, side of it, so were you surprised by the results that small copepods and large copepods [indistinct] better, than it did relative to the large copepods?
[Dave Kimmel] No, not surprised at all.

The algorithm is, it's sort of predictable. You can sort of tell by the images you give it how well it's going to do by sitting there long enough.

You know even if you go in and label, it's more difficult to label small copepods. So I'm struggling at it, so the algorithm itself is probably going to do the same. The question then is you know as you shift magnifications around how good is it at detecting small versus large, if you keep the magnification the same. So there's a lot of, there's subtle nuances in it. 

And so what I'm trying to do is create an algorithm that is as robust as humanly possible. Whatever you throw at it, it's going to go - oh yeah that's what I'm supposed to be looking at and counting. And you don't have to, you know, put in a lot in. I want this to be truly automatic. In other words I don't want it to be, to require a lot of front-end work by the individual to get the image just right. I want, you know, put the slop, slop the sub sample under the, whatever imagery you have, take an image, move on.
So... that's the hope anyway. But I don't know if we're there yet.
[Deana Crouser] We have one, chat, chat question from Michael.
Can you please go into more detail of how this data is used? For example, are the estimates of copepods used to inform recruitment of fish species and stock assessment models?
[Dave Kimmel] So yes. Some information is used. It goes into stock assessment models. For example the large Copepod Callanus in the Bering, can be used as a predictor for Walleye Pollock - H3 populations. So Lisa Eisner's published a paper on that. So they go in there they also go in risk tables, for example during the warm years in the Gulf of Alaska, we had very low zooplankton abundances during our spring surveys. 

And that was a bit of an alarm bell and it went into the risk table and they actually used it to reduce catch for some of the Walleye Pollock and other species based on system-wide productivity. So yeah they are used. As far as recruitment models, you know, they were used in the past and for example in the Gulf, Walleye Pollock recruitment has kind of shifted away from, from the larvae over to sort of a predation switch. So it's not as viable, but you know the system's changing, so I think we'll have to revisit that. Hopefully that answered the question.

Overlapping specimens.
Yeah, they overlap all the time. And the algorithm can deal with that. It's actually good, it once it learns how to detect, it can detect overlapping things. It's a bit more of a problem but it's like, I was talking about with the ZooScan, you know if the organisms can't touch it's a lot of time then you got to put in the individuals in there. 

You got to separate out the differences, you gotta, so we said we need an algorithm that's more robust than that and this algorithm that we've been using is, it's actually very good for not only, sort of microscope images but for benthic cameras, for acoustic data, it works really well, this algorithm. But yeah no problem with organisms touching, so far.

Sometimes it segments the same organism into two organisms, that's a problem.

Anything else on there?

[Emily Lamagie] How much data do you typically collect a season at the range of the models improving?

Do you have an outlook for getting to the point that you want to be at?

[Dave Kimmel] Yeah that's a great question. 
So for the annotated data set, we're producing, Poland is helping us to the tune of about 4,000 individual images a year that contain multiple individuals. And then somebody has to label them. That somebody so far has been me, so my capacity is somewhat limited but I think in our next agreement we'll talk about having our Polish colleagues help us label the images as well.
In that case I'm hoping once we have all the field images from this year, let's say we generate a couple hundred RZA images, this year coming year, and we use that in the training, I'm thinking by next year we could hopefully have an algorithm that's semi-ready for prime time. If Julie Keister is listening, don't put that in my performance plan please.
[Emiliy Lamagie] So, we have time for one last question. If anyone wants to jump in in the chat or in the room.

Oh we got one from the chat.
Can you tie the abundance counts to satellite data. Which is ocean color?
[Dave Kimmel] I've seen people attempt that over the years and I have yet to be convinced that it is possible. I've seen some work that looked at broad-scale satellite imagery related to CPR counts and other things. So far I'm not convinced, is the best way to put it, but I haven't tried it myself.

[Emily Lamagie] And Julie commented "ha ha".
[Laughter]

All right, thank you. Let's give a round of
applause to both of our speakers today.

[Applause]

And then we'll see you back next week for another EcoFOCI seminar. That will be our last one for the season.
[Dave Kimmel] I think we're done.

Thanks!