Saturday, February 10, 2024

ISIS and ASP on Windows 11 (WSL, take 3)

Based on this original post for Windows 10 and with help from M. Phillips (LPL, see: alternative instructions and demos), I will update this post to get ISIS and ASP installed on Windows 11 (again using using WSL). Warning: this requires a lot of downloading. This took about two hours on my laptop and decent wifi, but I did have some issues with the data download.

First, ASP = NASA Ames Stereo Pipeline (GitHubDocs)
ISIS = USGS Integrated Software for Imagers and Spectrometers (GitHub, Docs)

(1) get Linux for Windows 11
I recommend Ubuntu 22 LTS installed via the Windows app store (or via wsl powershell command-line, "wsl --install -d Ubuntu-22.04").

And if you don't have access to the MS App store, you can try a more manual install.

(2) For Windows, install Xming and launch (it runs in the background minimized)

(3) Once running Ubuntu, update it (this just gets your Ubuntu system up to date)
> sudo apt-get update
> sudo apt-get upgrade

(4) Now and let's get X11 and GUIs working:
> sudo apt install -y xclip gnome-themes-standard gtk2-engines-murrine dbus dbus-x11 x11-apps libgl1-mesa-glx

Looks like Anaconda wants even more (I consider this optional for now):
> sudo apt-get install libgl1-mesa-glx libegl1-mesa libxrandr2 libxrandr2 libxss1 libxcursor1 libxcomposite1 libasound2 libxi6 libxtst6

-- test X11
-- add this line to bottom of .bashrc (or type):
> echo export DISPLAY=:0 >> ~/.bashrc
> source ~/.bashrc
> xeyes

(5) Get Anaconda (or miniconda should work too).
> wget https://repo.anaconda.com/archive/Anaconda3-2023.09-0-Linux-x86_64.sh
-- or get latest version from download page: https://repo.anaconda.com/archive/ 
> bash Anaconda3-2019.03-Linux-x86_64.sh
-- follow install instructions and allow installer to update your shell (~/.bashrc) when it asks (will need to type "yes").
-- once installed, restart the Ubuntu terminal from Windows' powershell (typing wsl) or simply from Windows start.

(6) Get ISIS and ASP applications
-- note for ISIS only, follow the installation pages on GitHub.
or these steps:
> conda create -n asp3
> conda activate asp3

> conda config --env --add channels conda-forge
> conda config --env --add channels usgs-astrogeology
> conda config --env --add channels nasa-ames-stereo-pipeline

> conda install  -c nasa-ames-stereo-pipeline -c usgs-astrogeology -c conda-forge stereo-pipeline==3.3.0
//This can take a while. If you need a more recent version, remove the ASP version above. By locking to ASP v3.3.0, it looks like ISIS v8.0.0 will be install alongside. Also mamba is a much faster environment solve, if you install it.

-- Finally, execute the ISIS variable initialization script with default arguments. execute the ISIS variable initialization script with default arguments.
-- this script prepares default values for: $ISISROOT, $ISISDATA, $ISISTESTDATA
> python $CONDA_PREFIX/scripts/isisVarInit.py

(7) Get ISIS3 base data
> conda activate asp3
-- we will need to get the base data, has target SPICE kernels and a handful of global DEMs for orthorectification. Feel free to delete any data you wont need.
> downloadIsisData base $ISISDATA

Data Notes:
--Due to what I can only call an unfortunate quirk for the files on S3 cloud (doesn't support symbolic links), you will get many copies of the same DEMs (Mars, Moon, ...). It is more than 12GB of extra files. Feel free to delete all but the last version under $CONDA_PREFIX/data/base/dems
--I ran out of space on my main drive, so I edited my environment variable "ISISDATA" to point to a USB drive "/mnt/d/data/". This can be updated here "~/anaconda3/envs/asp3/etc/conda/activate.d/isis-activate.sh"
    --To get Ubuntu to see a new drive, you may need to get out and back into Ubuntu. In a powershell you can type "wsl --shutdown" and then restart with "wsl".

Done. at least for the minimal install. You may need lots more SPICE data for a particular mission, read more about that on the ISIS install page.
-- So the next time you launch an Ubuntu terminal, just type "conda activate asp3" to run ISIS and ASP routines.

Thursday, January 6, 2022

Tips to interact with Astro’s WMS maps

update from the original 2014 post:

For years we have supported our live mapping services (called Web Mapping Services, WMS) for use within our own web mapping tools but also for the community to use. For example, these available WMS layers are viewable from our Planetary Nomenclature, PILOT, CartoCosmo viewer, etc (example Aram Chaos on Mars). Thus they are ideal for use in web mapping apps like OpenLayers and Leaflet and also GISs like QGIS, ArcMap, or ENVI. And you can even build custom images from a simple browser call or using apps like GDAL (gdal_translate). Fortunately, most bases used for our WMS services are available for download from Astropedia https://astrogeology.usgs.gov/search. And for direct use in ArcGIS Pro and ArcMap, many are also listed from the Esri GIS portal. Note that many of the latest WMS Tiled layers in the ESRI portal, as served by ESRI, are appropriately tagged for the body they represent (no projection workarounds as described next).


Projection quirks. Because of the original limitations for WM* services as defined by the Open Geospatial Consortium (OGC), most services we support are tagged as EPSG:4326. This code defines Earth (specifically WGS84) in degrees. Fortunately for mapping services, degrees are degrees on most mapped bodies. This is our current (unfortunate) workaround to support these services across as many applications as possible for interoperability. 

So to use these correctly for measures, not just for viewing, it will require the user to know how to override this EPSG code in the application. Generally, using these services in packages like ArcMap or QGIS simply requires the user to first set a correct planetary-based map projection and the WMS layer will be "automagically" project correctly (see more below). Also for OpenLayers, Cesium, or Leaflet, you can override the internal definition of EPSG:4326 to define the correct radius. For example, see CartoCosmo or an updated version of Cesium.

The community has been working on this issue for a while. In 2021, we revised a new set of OGC codes (IAU2015) for the planetary domain (see this abstract). These are now available as built into the PROJ library (and GDAL) and will hopefully be pushed out into some other applications (e.g. Mapserver, QGIS, etc.). You can see these newer codes in action here. The original IAU2000 codes, as proposed in 2006 (abstract), were even incorporated in Lunaserv (and also a fork of MapServer but it never pushed back into to main).

For Polar map-projected services (in meters), you will also need to make sure your application correctly overrides the service using a Polar Stereographic projection (making sure body (radius) is defined correctly). All our polar WMS services are correctly using meters on the defined planetary body (even if the code is defined as Earth).
  

Example uses:

For a simple interactive image request, again from any layer listed above you can type this into any browser and modify it at will. HTML example (just click to test):
http://planetarymaps.usgs.gov/cgi-bin/mapserv?map=/maps/mars/mars_simp_cyl.map&SERVICE=WMS&VERSION=1.1.1&SRS=EPSG:4326&STYLES=&REQUEST=GetMap&FORMAT=image%2Fjpeg&LAYERS=MDIM21&BBOX=221,15,231,25&WIDTH=1000&HEIGHT=1000
To change the request, pay attention to these keywords:
  • LAYERS: use listing from the page above
  • BBOX: for LonMin, LatMin, LonMax, LatMax
  • WIDTH and HEIGHT for number for the number of pixels (equates to final resolution).
  • more examples: Mars THEMIS DayMars MOLA, Lunar WAC, Io Galileo SSI ,Mercury Messenger
  • Nomenclature layers were changed from a deafult "0 to 360" Longitude system to a "-180 to 180" Longitude system in Jan. 2023. Let us know if is a problem. 
  • Nomenclature -180 to 180 Longitude: example Mars (&LAYERS=NOMENCLATURE_180) 


For GDAL see our tips page for installing and the GDAL wms page for more (also related). See the xml text below (or on Git) for a Mars MDIM21 example (again customize the address and layer names from the listing above). Using the XML block you can run this below:
> gdal_translate -of PNG gdal_MDIM21.xml mdim21_out.png

File: gdal_MDIM21.xml
<GDAL_WMS>
  <Service name="WMS">
    <Version>1.1.1</Version>
    <ServerUrl>http://planetarymaps.usgs.gov/cgi-bin/mapserv?map=/maps/mars/mars_simp_cyl.map</ServerUrl>
    <SRS>EPSG:4326</SRS>
    <ImageFormat>image/jpeg</ImageFormat>
    <Layers>MDIM21</Layers>
  </Service>
  <DataWindow>
    <UpperLeftX>150</UpperLeftX>
    <UpperLeftY>45</UpperLeftY>
    <LowerRightX>160</LowerRightX>
    <LowerRightY>35</LowerRightY>
     <SizeX>1000</SizeX>
     <SizeY>1000</SizeY>
  </DataWindow>
  <Projection>EPSG:4326</Projection>
  <BandsCount>3</BandsCount>
  <DataType>Byte</DataType>
</GDAL_WMS>

Using IAU_2015 codes in PROJ (for GDAL 3.4+), override the projection (so it is correct).
> gdal_translate -of PNG -a_srs IAU_2015:49900 gdal_MDIM21.xml mdim21_out.png

Note code 49900 is a spherical Mars in degrees. To expand it, type: gdalsrsinfo IAU_2015:49900

In QGIS v2.x you can add a "WMS" layer and copy this example "get capabilities" linked in from the page above. Then you can pick one or more of the available layers:
http://planetarymaps.usgs.gov/cgi-bin/mapserv?map=/maps/earth/moon_simp_cyl.map&service=WMS&request=GetCapabilities


By default, if added first, these layers will think they are on Earth. But if you define the QGIS project using the built-in planetary projections for the Moon, Mars, or other and allow projection-on-the-fly, they will behave appropriately.


For QGIS v3.x, there are newer projection checks so you must override the original EPSG:4326. First, add in the layer accepting the incorrect EPSG:4326 code. Once loaded, open the WMS's layer Properties and you can override the projection. In the Properties window, under the Source tab, type in Mars or Moon, etc., and override the projection. It should now be correctly defined and usable. See the image below. Another tip is to define a custom map projection in meters (under Project Properties, CRS tab). For example, 


Another QGIS v3.x tip is to define a custom map projection in meters for the project. Under Settings, Custom Projections, define a new planetary projection you want to use (see next image). Now under Project Properties, CRS tab, set this project with your newly defined custom projection. Using a meters projection in QGIS v3.x can help with calculated measurements (e.g. Ellipsoidal measurement tool).



ArcMap and ArcGIS Pro are actually more forgiving with projections. But it is still best to first add a layer or image that is correctly defined. This will set the current project (or view) to that planetary projection. Now once you add in a WMS layer, it will (usually) conform to that defined map projection set in the view. For our polar WMS services, you will want to set the map projection to match Polar Stereographic (+-90 central lat, in meters) on the body you are using.

Lastly, using our planetary extensions within OpenLayers, you can easily make your own web maps using these same services. Copy and edit this example html document with the layers from above which uses our own planetary-ready version of Open Layers. There is also CartoCosmo, which is a student-created planetary-ready version of Leaflet (based on code from our original OpenLayers version). Once tweaked, feel free to host on your own website.


Hope this helps,
Trent

Saturday, March 30, 2019

LPSC 50th – Tech Lunch notes, March 20th, 2019

What and Where: In 2019, at the 50th Lunar and Planetary Science Conference (LPSC), several participants gathered to discuss planetary technology-based topics. This meeting was held over lunch at the restaurant in the conference center and lasted about 1.5 hours. This is the 3rd or 4th such technology lunches held at LPSC. The unofficial tag-line for this meeting was “this is a no science zone”.
Why: LPSC is generally heavily science focused for talks and discussions, thus there are not many opportunities to discuss planetary technologies as a group. We note that abstracts and poster sessions are an excellent method to present technology topics, but again a larger group interaction is missing.
Who: The initial email invite went out from Trent Hare (Astrogeology) to essentially a random list of community members known to support various technology initiatives. Several others were able to attend from word of mouth. In total about 25 participants were able to attend. Facilities represented included NASA facilities (e.g. AMES), JPL (e.g. NAIF, PDS nodes, MIPL), USGS (Astrogeology), RPIF, APL, PSI, and many from U.S. and international universities (mission teams and students).
How: Due to the size of the group, the main topics discussed were guided. One of the main intents was to allow participants to see what others in the room are working on to hopefully spur on future collaborations. If you find a topic you are interested in, please reach out to share ideas, methods, software, and hopefully increase partnerships.

Topics:
Quick introduction for new Astrogeology community contacts:
·        Robin Fergason (rfergason@usgs.gov) has taken over as lead technology manager for Astrogeology. Please contact her for any comments or issues related to any services Astrogeology supports.
·        Jay Laura (jlaura@usgs.gov) took over as software lead at Astrogeology (including ISIS3). Jay introduced several community software initiatives at Astrogeology as we move to a more open source initiative for ISIS3 and other supported software tasks. https://github.com/USGS-Astrogeology. GitHub issues are being used for all projects. We want to hear about issues, feature requests, documentation improvements, etc.
o   To facilitate this, Astrogeology software will also have oversite from a Technical Steering Committee (TSC) for several of their projects. For more see: https://github.com/USGS-Astrogeology/TSC.
o   New discussion forums are available. https://astrodiscuss.usgs.gov. Sign-in using a github acnt.
o    Please follow the Astrodiscuss to see updates and community Requests for Comments (RFCs). These will also available at the ISIS3 Wiki: https://github.com/USGS-Astrogeology/ISIS3/wiki.
o   Note ISIS3 is also moving to a new versioning system. Thus, be prepared that version 4 is coming. This updated versioning better follows industry standards for software.
Image Matching and Change Detection
·        The first question posed to the group was, “how many are interested in image matching techniques”. About 60% or more of participants raised their hands. This obviously has wide uses across several applications including automating image-to-image registration methods, image to “ground” registration (tying one image type to another image type or tying to a controlled base), terrain generation using stereo-pairs, facilitating change detection, etc.  
·        There are lots of papers and research in this topic. Simple searches provide many of the initial results below thus consider this a sampling of available resources (please send me more to add or include below in comments):
o   Multi-instrument coregistration: https://ieeexplore.ieee.org/stamp/stamp.jsp?arnumber=8077748
Machine Learning (and Feature Extraction)
·        While not as popular as image matching, ML topics were also popular among participants. This is probably one of the fastest growing research areas for planetary (and Earth-based) domains. Like above, a simple search provides many papers and abstracts (please send me more to include or add in the comments below):
o    Machine learning session at 2018 AGU: https://agu.confex.com/agu/fm18/prelim.cgi/Session/50013
o    Lots of automatic crater detection papers!
NAIF and SPICE
·        Chuck Acton stated the transition to a multi-thread SPICE library is proceeding well.
·        Syncing SPICE kernels remains to be a large community issue. For example, if SPICE kernels are updated, how are they to be released? Sent back to NAIF? And how and who reviews them? As more groups are creating controlled mosaics, this issue will continue to grow.
·        Cosmographia continues to be supported by NAIF. One idea stated by the group included making SPICE kernels more easily integrated into Cosmographia. For morehttps://naif.jpl.nasa.gov/naif/cosmographia.html
·        SpiceyPy, supported by Andrew Annex, has recently been updated to v2.2.0. https://github.com/AndrewAnnex/SpiceyPy.
·        There is an upcoming (free) SPICE training (June 4-6, 2019). I believe all materials will be made available to all. To attend, please register: https://naif.jpl.nasa.gov/naif/WS2019_announcement.html
·        While Astrogeology attempted to use Cosmographia for ISIS3’s control environment, it has been deprecated.
Planetary Web Mapping Services (WMS)
·    Lunaserv is still being updated. It is the first WMS server to support Planetary codes. http://lunaserv.lroc.asu.edu/about.html
·    There is a new branch of Mapserver by Jean-Christophe Malapert which also uses planetary codes. It is not yet pushed back into Mapserver baseline. For more information, contact Jean-Christophe on github. 
Planetary Data System (PDS)
·      There was a strong attendance of PDS personnel at LPSC.
·     Thanks to Nick Estes’ idea, we have an initial method to archive GIS data in the PDS. We are using a variant of GeoCSV with a PDS4 label: https://giswiki.hsr.ch/GeoCSV. An early implementation is now in GDAL for testing: https://www.gdal.org/frmt_pds4.html
·      Upcoming technical session for PDS personnel in April will cover many topics listed here and including PDS4, tools for PDS4, facilitating search, using Github, long-term roadmap, etc. Supporting the community and better advertising updates and direction for the PDS is also a main goal.

Upcoming Technical Workshops for the Planetary Domain
·   The 4th Planetary Data Workshop (not free), June 18-20, Flagstaff, AZ: https://www.hou.usra.edu/meetings/planetdata2019/. It will have presentations, posters, tutorial demos, and discussions.
·   The 2nd Planetary Mapping and Virtual Observatory Workshop, July 1-3, Domaine de St. Paul, Saint-Rémy-lès-Chevreuse, France. This a great “partner” workshop for the Planetary Data Workshop and covers many related technical topics. It will have presentations, discussion workshop, and tutorials. https://epn-vespa.github.io/mapping2019/
Recent Published Books (many technical chapters available)
·  Planetary Remote Sensing and Mapping: https://www.taylorfrancis.com/books/9780429000515
·  Planetary Cartography and GIS: https://www.springer.com/us/book/9783319628486

Saturday, March 2, 2019

JPL Geospatial Technology Round Tables, Feb 11-13, 2019


Table of Contents

Introduction

Between February 11-13 a series of discussion style round tables were held at the NASA Jet Propulsion Laboratory (JPL) in Pasadena, CA. The meetings were hosted by the Instrument Software and Science Data Systems Section 398, and largely represented by members of the Multimission Image Processing Lab (MIPL) and Planetary Data Systems (PDS) Imaging Node, although others were able to join the discussions in person or via webex. The Astrogeology Science Center (ASC) of the U.S. Geological Survey (USGS; Flagstaff, AZ) was represented by Trent Hare and ASC software developers during the camera model discussions. These discussions were initiated by Dr. Thomas Logan and Paul Ramirez and who also provided travel support for Mr. Hare.


These notes are a summary/overview of the discussions as prepared by Mr. Hare, with minor changes provided by JPL participants (R. Deen and T. Logan).


Planetary and Earth Data Format Interoperability, Mon Feb 11

We began the discussion by listing many of the formats commonly used in the planetary (and Earth) community. The formats included: VICAR, ISIS3, PDS4, FITS (GeoFITS), GeoTiff, Jpeg2000, NITF, etc.

·       Jpeg2000 was discussed to help understand its limitations and its seemingly decline in popularity (although it still is used within the NITF format). Issues listed included: 1) while current open source implementations are available, they are not very robust, 2) the popular encoder/decoder Kakadu, while it used to be open access, continues to be locked down, and (3), while 32bit floating point is defined in the specification, it is not widely supported. Jpeg2000 is currently allowed in PDS3 archives (HiRISE and LOLA archives), but is now banned from PDS4 archives. This is due to PDS4 archives only allowing raw binary streams, thus no compression of any style is allowed.
·       Both Video Image Communication And Retrieval (VICAR) and the Integrated Software for Imagers and Spectrometers (ISIS3) formats are critical for the processing environments and missions they support. Changing formats would be difficult for either to support. Because VICAR uses an in-line label followed a binary stream, it is recognized that a simple reader could be written. Also, VICAR meets the PDS4 criteria to allow a VICAR image to be archived (as-is) along-side a PDS4 label which describes how to skip the in-line label.  Because ISIS3, by default, supports tiled data streams, reading and supporting PDS4 archives is a little tougher (note that ISIS3 also support raw binary streams). Also, VICAR does have limited compression support throught it’s Automated Fusion of Image Data System (AFIDS) branch of VICAR. The main branch of VICAR at JPL is used for in-situ missions like InSight, Mars2020, and other planetary mission activities.
·       The AFIDS team has implemented a robust reader/writer in the Geospatial Data Abstraction Library (GDAL) for the VICAR format (which is available in the current Open VICAR distribution).  However, it has not been pushed back into GDAL’s baseline code release. This is a stated goal for the group. There is already a limited VICAR driver in GDAL (by Sebastian Walter, Freie University Berlin), which needs to be taken into consideration prior to merging the AFIDS version.
·       GDAL does have read support for PDS3 and the older ASC ISIS2 format. ASC has funded support for a basic ISIS3 reader and writer. Lastly, GDAL does have basic support for PDS4 but requires a well-defined template label to make it ready for archival.
·       The FITS format now has a proposed and implemented extension, called GeoFITS, allowing for more standardized geospatial map projection definitions. GDAL contains the first implementation as written by Chiara Marmo (University of Paris-Sud). It is still new and not yet widely used. Reference: Marmo, C., Hare, T. M., Erard, S., Minin, M., Pineau, F.‐X., Zinzi, A., et al. (2018). FITS format for planetary surfaces: Definitions, applications, and best practices. Earth and Space Science, 5, 640–651, https://doi.org/10.1029/2018ea000388.
·       The JPL-created and now Esri-supported format, called the Meta Raster Format (MRF) by Lucian Plesea, was discussed for its ability to optimize streaming services like Web Mapping Services (WMS). It is analogous to a file-based database since it supports an index into packed image tiles. It supports tiles as compressed by JPEG, PNG, the new Limited Error Raster Compression (LERC) to name a few. LERC, originally written by Thomas Maurer, is an open specification by Esri and an open source implementation is available within GDAL. Because it can be lossless or lossy (with user defined precision), it has the potential to take on compressions like LZW and even Jpeg2000.
·       The group seemed to agree that the GeoTiff format is the current best choice for interoperability between applications. As a free and open source library, GDAL also maintains the reference implementation. GeoTiff is well supported within AFIDS (a VICAR branch) and the NASA AMES Stereo Pipeline (ASP), but for other planetary applications (like ISIS3), the GeoTiff format must be converted into their native format. Using a raw (uncompressed and untiled) GeoTIff allows one to use the format in a PDS4 archive. Actually, in GDAL, when converting to a PDS4 image, GDAL optionally allows one to store the output image within a properly labeled GeoTiff (with a PDS4 label pointing into the GeoTiff). Lastly, it was noted that several projects like the Lunar Mapping and Modeling Project (LMMP) and related Trek projects, the Phobos Working Group, and the ASC-supported PDS Annex portal all prefer GeoTiff as their primary format (optionally with a PDS label).
·       In the process of discussing image file formats, a number of useful file attributes were identified. These include: 1) Support for multiple bit structures such as 8-64 bit integer, floating point, real and imaginary; 2) Open source (100% non-propriety); 3) Support for Compression; 4) Support for tiling; 5) Support for Georeference; 6) Straight-forward binary layout; 7) Straight-forward label design (possibly integrated and/or detached); and 8) Support for multiple bands.

Collaboration Topics

·       The group agreed that we are not all going to move everyone over to one format. There is no “killer” format or “one format to rule them all”. Thus, it was agreed that supporting GDAL seemed to be the best method to help support interoperability across applications. One collaboration idea would be a test suite providing results in GDAL for these conversions. For example, converting VICAR to GeoTIff back to VICAR and document what is lost. This could be applied across many of the popular formats like ISIS3 to VICAR and back to ISIS3. If too much information is lost, potentially teams could update the drivers in GDAL to help. Also supporting a side-car metadata file during conversion might also help. This metadata side-car file is already supported in GDAL, but the drivers would need to take advantage of it.
·       The AFIDS group agreed to find resources to push their current in-house GDAL-based VICAR reader/writer back into the public release of GDAL. This would require assessing the impact to the current VICAR driver already in GDAL.

Open Source Software Issues, Tue Feb 12

  • Discussions in this topic were not about the benefits of open source as all persons and projects represented have goals for making their code open source. Mostly this discussion focused on perceptions on open source projects and ideas around supporting open source projects.
  • In 2015, Bob Deen (JPL) helped make a majority of VICAR open source. However, there are parts of VICAR that still have not been released. These include: 1) code which contain ITAR constraints; 2) code that uses licensed software (notably Numerical Recipes); 3) parts of the in-situ code-base; and 4) the AFIDS branch of VICAR. Besides the ITAR constraints, there are goals to open source both the in-situ and AFIDS versions.
  • For ASC, all products, including software, are mandated to be public domain. Only recently though has ASC pushed their code-base to Github and will strive to “behave” like an open source project (see below for more).
  • The image processing package Opensource Multi-INstrument Analysis Software (OMINAS), created by Joseph Spitale, is now mainly maintained by PSI and JPL, has also stated plans to open source their code. OMINAS is built on the commercial application IDL, currently owned by Harris Geospatial Solutions. The OMINAS maintainers currently support a free run-time for IDL and their code which users can take advantage of without an IDL license.
  • It was discussed that releasing one’s code can be difficult for all members of a team when it is perceived as a specialized capability that potentially maintains future funding opportunities. Within the group, most stated that the benefits and importance of releasing their code outweighed this potential risk.

Collaboration Topics

  • The group ended with discussions on what it is to be an open source project and ideas on how to fund it. It was pointed that the key behind successful open source projects is your user community. While releasing the source code is a great first step, if there is no support to foster and work with your user community, the project probably won’t make it for the long-term. Thus, calling out resources within your funding opportunities to foster your community and fund other open source projects you rely on is viewed as an extremely important aspect for any open source project.  This includes the ability to take community input, whether they are bug reports, new ideas, or even the ability to accept user supplied code into your project (granted it must meet your coding standards).
  • Since we all support the same or closely tied communities within both the planetary and Earth-based domains for image processing, we should continue to discuss how to better support our user community across our different applications. For example, allowing them to interoperate (e.g. formats, camera models, shared capabilities), would be a great step forward.
  • A good resource for this topic includes this free publication: National Academies of Sciences, Engineering, and Medicine. 2018. Open Source Software Policy Options for NASA Earth and Space Sciences. Washington, DC: The National Academies Press. https://doi.org/10.17226/25217.


Talk: Activities at the USGS Astrogeology Science Center, Mon Feb 11

  • presentation: available from dropbox.



“Big Data” Streaming Technologies and GeoSpatial Data Portals, Tue Feb 12

This round table covered several ongoing projects at JPL which are tackling these big-data topics. Like JPL, ASC has also been supporting streaming methods since the late 90s. Several projects were discussed including:

  • Science Data Analytics Platform (SDAP). “SDAP is a technology software solution currently geared to better enable scientists involved in advancing the study of the Earth's physical oceanography”. As an Apache incubator project, it has been an open source project from the start.  While centered around oceanography data, many of the core capabilities could be used in other projects. Instead of the generation of derived products on-disk, they are working towards real-time calculations. These products can be recalculated based on an encoded URL. The goal is to try to limit the need to large mosaics to be downloaded. If results must be provided, try to summarize results as small tables. Again, the encoded URL can be used to recreate the derived maps.
  • The Global Imagery Browse Services (GIBS) are a set of standard services to deliver global, full-resolution satellite imagery in a highly responsive manner. This project is riding on technologies created at JPL and use OGC web standards including Web Map Tile Service (WMTS) and the similar Tiled Web Map Service (TWMS). Most all of their code is made available from within GDAL or from their Github site.
  • There was some discussion on the JPL Trek system. Like GIBS, Trek builds on many of the same standardized services and Open Geospatial Consortium (OGC) technologies. The interface they support, and have extended for planetary, is Cesium. CesiumJS is an open-source JavaScript library for 3D globes and maps as founded by Analytical Graphics, Inc.
  • ASC has been recently working with Esri and Caltech to test Web serving a couple of extremely large mosaics including the Caltech Mars CTX mosaic (~1TB in size, lossy, at 5m/p) a USGS- and Esri-create uncontrolled Mars HiRISE mosaic (19TB in size, lossy, at 0.25 m/p). These should be released by March 2019.

Collaboration Topics

  • The grouped talked about goals of providing cloud services (applications) next to these cloud-provided data sets. They also discussed how such systems can be maintained for the long-term. For example, how are researchers able to recreate their results if the service is no longer provided. This is an area for rapid growth and support as data sets continue to get larger.




End-to-End Processing of Planetary Data, Camera Models to GIS Products, Wed Feb 13

While this round table provided time for both camera model discussions and GIS products, largely we discussed the Camera Sensor Model (CSM) standardized API. ASC provided an introduction to the CSM standard. In short, the CSM API was conceived and implemented by the U.S. Air Force to standardize the use of camera models across different photogrammetric applications. The CSM standard provides a standardized API only, not the actual implementation of the camera model. It is now maintained by the CSM Working Group, which has several government, university, and industry members. The CSM is currently used by several photogrammetric applications including commercial applications like BAE’s Socet GXP, Harris Geospatial Solution’s ENVI, and Hexagon’s ERDAS Imagine. ASC is currently working on CSM support in ISIS3 and is helping to support the CSM within NASA’s Ames Stereo Pipeline (ASP).
·       ASC is currently working both a CSM open source framing and a pushbroom (line scanner) implementation. The pushbroom was originally developed by BAE and updated by ASC. With support from BAE, it is now released on Github using a permissive BSD license. ASC in collaboration with the CSM Working Group, the 3.0.3 API supports variable radii for planetary applications. Previously it was locked to WGS84 (Earth).  ASC has also created a Simplified Wrapper and Interface Generator (SWIG) for the CSM library which provides Python access to the API. It can also provide access to JAVA, Ruby, and many other languages. Note that ASC supported camera models in ISIS3 (more than 50 instruments) are currently developed using a unique in-house design, but they are working on supporting CSM implementations also.
·       The AFIDS Team provided background for their software called “Mars Nest”. The software is a georeferencing pipeline that automatically co-registers most two map-projected images to each other. For unmapped (EDR) images, they have developed procedures to use a Replacement Sensor Model (RSM) as a close approximation for a rigorous camera model. Current implementations use the NITF format to store the RSM metadata (called Tagged Record Extensions or TREs), which could potentially be stored inside a PDS4 file. Similar to a CSM, an RSM implementation is another method to provide interoperability across photogrammetric applications. 
·       The CAHVOR camera model is used by several at JPL for in-situ and orbital cases.
·       The NASA Ames Stereo Pipeline (ASP) currently uses both ISIS3 available camera models and their own camera models mainly for Earth-based instruments. The ASP team is currently implementing capabilities to incorporate CSM.
·       Lastly, for GIS data products, we did have a short discussion from Caltech providing details for their recently released CTX near-global mosaic for Mars. Not only did they optimize the input data to remove hazy images, they also optimized their workflow to maintain the ability to map every pixel in the mosaic back to the input image. They have also been provided funding, via a PDART, to make an even better version (~2 years). Input from the community is welcome. The current version will soon be available as a streaming cloud service hosted by Esri.

Collaboration Topics


  • After the introduction to CSM, there were many questions about whether the standardized CSM API would be capable enough for current applications. While the relatively simple API does lock the user into the standard (for the sake of interoperability), the implementation is left up to the user. Thus, it should be able to support most applications (but more research or pilot studies should be done). It was also brought up that the CSM API doesn’t make the implementation of a camera model any easier. But by providing several open source implementations, there is hope it might help with adoption.
  • Specifically, for in-situ applications, there was discussion if the CSM would be beneficial. Within the planetary community, there has been a long-time goal for the better integration of orbital and in-situ mission data. By allowing both orbital and in-situ photogrammetric applications to share the same camera model should actually help to realize this goal. Of course, a collaboration across projects to pilot this would be needed.

Tuesday, January 8, 2019

Uncontrolled Global HiRISE Mosaic

Mars High-Resolution Tiled Web Services


linked from: http://bit.ly/HiRISE_mosaic

An Esri introduction and Story Map for these layers: https://www.esri.com/arcgis-blog/products/arcgis-online/imagery/observing-the-red-planet/

These new Mars service hosted by Esri highlights controlled "foundational" mosaics including Viking MDIM v2.1 [231 m/p], MOLA/HRSC Blended Hillshade [200 m/p], and the USGS-created THEMIS Daytime and Nighttime mosaics [100 m/p]. We have also included the preliminary uncontrolled but high-resolution mosaics created by the Caltech Murray Lab for CTX [5 m/p] and USGS-generated HiRISE PSP/ESP mosaic [0.25 m/p]. Note on disk the CTX is lossy compressed to 1TB (originally at ~5TB) and the HiRISE PSP/ESP is lossy compressed to 19TB (originally nearly 100TB). Datasets have been compiled together and gratefully hosted as a Tiled Web Services by Esri (using AWS cloud). Data descriptions and their creators are provided on the simple viewer page (and referenced below). For these services both CTX and HiRISE are dynamically stretched server-side. The CTX stretch applied is "0:0,64:32,192:224,255:255" and the HiRISE mosaic is stretched from 10-bit to an 8-bit range is defined as "0:0,1:1,1030:255".

As stated in our abstract, this is really a technology evaluation to push our current limits for available processing clusters and data storage capabilities. We also used novel raster formats (MRF) and compression techniques (10-bit Jpeg/ZEN), and optimized distributed cloud capabilities. All capabilities to create and serve the data are using open source software (e.g. GDAL) and open OGC standards.

See below for references and additional download data resources.

Access the layers:

please be nice: using this test server to scrape large swaths of data is not allowed. But use (view) the resources in your GIS or web GIS viewers all you want. Also there is no need to build statistics - this will bog down the application and the server. 
  • A simple web viewer to take the layers for a test ride (click "Open in Map Viewer")
  • list of many of the individual services.
  • For ArcMap and ArcGIS Pro, simply head to ArcGIS Online (also just called Portal). See animations below.


  • For other applications you can use the addresses found here (zip). Drag the *.til or *.dem into your Desktop GIS. Note these files are simply text files defining the tiled WMS address (crack open in a text editor to take a peak). Some of these are 16-bit (*.dem) and 10-bit services, thus desktop GIS applications should have no problem loading, but web viewers may not support some of these layers directly. Again - no need to build stats ever!
References

Plesea, L. and Hare, T.M. (2019). Uncontrolled Global HiRISE Mosaic, In Lunar and Planetary Science Conference (Vol. 50, pp. 1–2), The Woodlands, TX,  abstract #1986, PDF poster.

McEwen, A. S., Eliason, E. M., Bergstrom, J. W., Bridges, N. T., Hansen, C. J., Delamere, W. A., Grant, J. A., Gulick, V. C., Herkenhoff, K. E., Keszthelyi, L., Kirk, R. L., Mellon, M. T., Squyres, S. W., Thomas, N., & Weitz, C. M. (2007). Mars Reconnaissance Orbiter's High Resolution Imaging Science Experiment (HiRISE).  JGR-Planets112 (E05S02). Download map projected HiRISE Images.

Kirk, R. L. M., Holmberg, I. M., Keszthelyi, L. P., Redding, B. L., Delamere, W. A., Gallagher, D., Chapel, J. D., Eliason, E. M., King, R., & McEwen, A. S., Ultrahigh resolution topographic mapping of Mars with MRO HiRISE stereo images: Meter‐scale slopes of candidate Phoenix landing sites. (2008). JGR-Planets 113 (E00A24). Download HiRISE DTMs,

Dickson, J. L., Kerber, L. A., Fassett, C. I., & Ehlmann, B. L. (2018). A Global, Blended CTX Mosaic of Mars with Vectorized Seam Mapping: A New Mosaicking Pipeline Using Principles of Non‐Destructive Image Editing. Lunar and Planetary Science Conference (Vol. 49, pp. 1–2), The Woodlands, TX, abstract #2480. Download the data.

Fergason, R. L., Lee, E. M., Weller L. (2013). THEMIS Geodetically Controlled Mosaics of Mars, Lunar and Planetary Science Conference (Vol. 44, pp. 1–2), The Woodlands, TX . abstract #1642. Download the data.

Archinal, B. A., Kirk, R. L., Duxbury, T. C., Lee, E. M., Sucharski, R., Cook, D. (2003), Mars Digital Image Model 2.1 control network, Lunar and Planetary Science Conference (Vol. 34, pp. 1–2), abstract #1485. Download the data.

For more information see the simple viewer page.