MapServer

Project Status Report

FOSS4G-Europe 2015 - Como - 2015-07-15


Stephan Meißl stephan.meissl@eox.at @Schpidi

Thomas Bonfort tbonfort@terriscope.fr @tbonfort


Creative Commons License This work is licensed under a Creative Commons Attribution 4.0 International License.

What is MapServer?

  • Open Source platform for publishing spatial data to the Web
  • MIT-style license
  • Runs on all major platforms (Windows, Linux, Mac OS X)
  • OSGeo project
  • mapserver.org

MapServer History

  • 1994: Steve Lime implements Web-based Arc/Info AML generator
  • 1994-95?: Frank Warmerdam releases Shapelib
  • 1997: v1.0 (limited sharing)
  • 1998: v2.0 via Fornet website
  • 1999: v3.0 via UMN MapServer website
  • 2001: PostGIS support by Paul Ramsey in v3.6
  • 2002: GRASS Users Conference - Trento
  • 2003-05: Yearly MapServer Users Meeting
  • 2006: Foundation of OSGeo with MapServer as founding project
  • 2006+: FOSS4G Conferences
  • 2009+: C Tribe Code Sprints
  • 2012: Addition of TinyOWS & MapCache ⇒ MapServer Suite (v6.2)
  • 2013: v6.4
  • 2014: v7.0beta & 20th anniversary
  • 2015: v7.0 (beta2 released 07/2015)

Statistics

  • ~150k Lines Of Code
  • ~12k commits
  • ~100 contributors
  • Mailing lists
    • -users ~1800
    • -dev ~400
  • International PSC
    • 14 members
    • NA: 9 (US: 5, CA: 4)
    • EU: 5 (FR: 2, FI: 1, AT: 1, HU: 1)
  • 113 RFCs

MapServer 7.0

  • Main new features
    • UTFGrids
    • WFS 2.0
    • Heatmaps/Density Maps
    • Javascript/V8 StyleItem/Geomtransform
    • Unification of attribute/geometry filtering
    • Layer Level Compositing / Blending
  • Others
    • Text rendering re-factoring
    • GD Removal
    • Removed BITMAP labels
    • Layer Level Encoding
    • WCS 2.0

UTFGrids

  • Specification by MapBox for vector tiles
  • Implemented in a "Google Summer of Code" project
  • Interactivity important (rollover, events)
  • Supported by OpenLayers

WFS 2.0

  • Download Services for INSPIRE

Javascript (V8) StyleItem/GeomTransfom

  • Integration of the V8 JavaScript engine
  • Style geometries/features programatically rather than through class expressions
  • "Experimental mode": backward compatibility not guaranteed
  • 
    LAYER
        ...
        STYLEITEM "javascript:///path/to/my/file.js"
        CLASS # empty class required
        END
    END
                                
    
    //Make symbol size 14 or 7
      var size = shape.attributes.NAME.length > 10 ? 14:7;
    
      var style = "STYLE SIZE " + size;
      style += " SYMBOL 'circle'";
      style += " COLOR 255 0 0 END";
    
      //Return style to MapServer
      style;
                                

Unification of attribute/geometry filtering

  • Currently filtering done by MapServer
  • PostGIS, Oracle, OGR support native spatial filtering e.g.
  • 
    where st_intersects(the_geom, st_geomfromtext('wkt...'))
                                
  • Significant performance gain particularly in WFS queries
  • Simpler to use
  • Was blocking release of version 7.0

WCS 2.0

  • Implements available extensions
  • Passes all available CITE tests
  • Official OGC WCS 2.0 Reference Implementation

HeatMaps

  • Vector to raster processing pipeline
  • Interpolation and rendering parameters
    • Automatic scaling
    • Interpolation in color spaces RGB/HSL
    • Weighting per attribute/expression
  • Supports tile mode

Interpolation in HSL color space

Interpolation in HSL color space

Interpolation in RGB color space

Pure density

Setting a fixed scale

Layer Blending

Layer Filters

Text rendering re-factoring

  • Before naive implementation duplicated for each back-end
  • Integration of Harfbuzz library
  • Supports complex text layout e.g. for Arabic text
    • Complex text layout
    • Unprocessed characters
    • With bi-directional processing
    • With glyph shaping rendering letters according to context
  • Barely visible to the end user, but
    • Exact label placement (alignment/centering/letter spacing of text)
    • Possibility of using one font per language

Improving performance (notice the log scales)

Malayalam Example

Tibéto-Birman Example

GD Removal

  • Added back in 1995
  • Too high maintenance costs despite its usefulness in certain niches
  • Automatic fall-back to AGG + 8bit


Removed BITMAP labels

  • Truetype font included by default now
  • Supports many glyphs without external dependencies
  • A few use cases are not supported
  • Automatic fall-back in place

Layer Level Encoding

  • Solves encoding handling problems e.g. data sources in multiple encodings in a single mapfile
  • Data sources internally converted to UTF8
  • The mapfile has to be encoded in UTF8 now
  • All output generated by MapServer in UTF8

MapCache 1.4

  • Tiling Library with Apache, FastCGI, Nginx and Node frontends
  • Versatile
    • Multiple Cache Backends
    • Multiple Protocols
    • Tile Management Features: seeding, recompression, interpolation
  • Native code: Fast and Efficient

Standard Tile Addressing

  • TMS
  • WMTS (RestFull + KVP)
  • VirtualEarth QuadKeys
  • MapGuide
  • Google x,y,z

WMS

  • WMS-Tiled
  • Responds to untiled requests by assembling tiles
  • Can mirror a full WMS service: layer-by-layer, layer lists
  • Forwards GetFeatureInfo requests

(OGC) Proxy

  • Intercepts requests that can be served from cached tiles
  • Forwards other requests to other upstream server(s)

Miscellaneous

  • Caching HTTP Headers
  • "Auto" Expiry of old tiles
  • Error Reporting
    • Error Message
    • Empty Image
    • Status Code
  • Meta-Tiling
  • Watermarking
  • Max-Cached-Zoom: Upsample lower level tiles

Image Operations

  • Image data can be recompressed/optimized before being stored into a tile
    • Avoid double JPEG compression, i.e. request PNG from source WMS
    • Be agressive on compression when storing a tile (e.g. quantized PNG with max compression)
    • CPU time for dynamic recompressions can be lowered, at the cost of more bandwidth

Image Formats

  • PNG
    • Compression level
    • Quantization (color reduction to 8bit palette)
  • JPEG
    • Compression level
    • RGB/YCbCr
  • Mixed (JPEG + PNG)
    • Use for raster coverages on limited areas
    • PNG for tiles with transparency
    • JPEG otherwise

Handling Empty/Uniform Tiles

  • Strategy dependent on the cache backend
    • Symbolic linking
    • Reconstruct Paletted 1bit PNG from uniform color
    • For fully seeded caches, treat absent tile as fully transparent

Grids

  • Multiple Grids per Tileset (e.g. WGS84 + Mercator)
  • Handles Grid Aliasing
    • EPSG:3857/EPSG:900913
    • Non Standard Codes
  • Grid Subsets
    • min/max resolution
    • Restricted extent (with tolerance!)

Caches

  • Backend able to store tile data for given x,y,z
  • Extensible: Exists(), Get(), Put(), Delete()
  • Some backend specific hacks: symlinking, blank detection

Disk Cache

  • Mainly for dev, testing, or small tilesets
  • Pros
    • Simple, just provide a base directory
    • Fast
    • Supports blank tile detection
    • (Re-)Use custom layouts: TileCache, ArcGis, Worldwind, Template
  • Cons
    • Difficult to manage large number of files
    • Hits File System Limits
    • Wasted space due to File Sytem block size

Sqlite Cache

  • Stores tile data as blobs in an sqlite file (/path/to/{grid}/{tileset}/{z}.db)
  • Can be split into multiple db files (per dimension, grid, tileset. Configurable max number of tiles per db)
  • Pros
    • Single File
    • Support any schema (MBtiles, custom...)
    • Efficient Space (blank tile detection possible)
  • Cons
    • Some tweaking needed for very large caches (more than 1 Tb)
    • Concurrent Insertions are slow

MemCache

  • Ideal for Temporary Data: Forecasts, Sensors, ...
  • Pros
    • Easily distributed
    • Automatic pruning
  • Cons
    • Limited Storage

REST Caches

  • REST Back-end
    • Support GET/PUT/DELETE for storing tiles
    • Authentication/authorization plug-ins for S3, Azure, and Google Cloud Storage
    • Costs highly depend on the use case (number of applications, bandwidth)

(Geo)TIFF Cache

  • Specialized storage for satellite imagery basemaps
  • JPEG tiles are read directly from a constrained (Geo)TIFF file hierarchy
  • Configurable number of tiles per TIFF file
  • Templated access to file name
  • Pros
    • Efficient on disk usage (TIFF container)
    • TIFF files can be shared with other applications
  • Cons
    • Limited to JPEG data (no transparency)
    • No Concurrent writing to a single TIFF file
    • Updates/deletion of individual tiles does not reclaim storage space
  • Use-Case

Misc Caches

  • Riak (distributed, extensible, fault tolerant)
  • Berkeley DB
  • Couchbase (soon)
  • Redis (soon)

Composing Caches

  • Composite Caches Select different caches based on rules (zoom level, grid)
  • Fallback Caches
  • Multitier Caches mix caches with different performance/storage characteristics. If a tile is not found in the fast/small/expensive/transient cache, it will be looked for in the slow/large/cheap/permanent cache and moved into the first one for subsequent accesses.

Seeder: Pre-generating tiles

  • Multi thread/process seeding
  • "Drill-Down" mode to take advantage of caching on source WMS
  • Seed particular subset
    • Given Dimension
    • Regenerate old tiles
    • Restricted min/max zoom and/or extent
    • Restrict to geometry (e.g. seed to level 18 all areas where population density is over a given value)
  • Pruning mode

Dimensions

  • Multiple Dimensions per Tileset
  • E.g. Forecast Time, Elevation, Client ID
  • Multiple Types
    • List of Values
    • Intervals
    • Regular Expression
  • Use-Case: BoM MetEye
  • Coming soon: Tile assembling when multiple dimension values passed

Time Dimension

  • Each timestamp corresponds to a limited extent
  • Dynamic Mosaic given a requested time interval
  • Create Animated GIF from selected tiles (soon)

Documentation translation

  • Started to use transifex for simple collaboration
  • Attract further translations

How to Contribute?

  • Users
    • Beta and release candidate testing
    • Bug reports on Github, including test cases
    • Documentation (wiki, transifex, Github pull requests)
    • Answers on the mapserver-users list and IRC
  • Developers
    • Bug fixes/patches
    • New features
    • Binary builds
MapServer

Project Status Report

FOSS4G-Europe 2015 - Como - 2015-07-15

schpidi.github.io/foss4g-e_2015_mapserver_status


Stephan Meißl stephan.meissl@eox.at @Schpidi

Thomas Bonfort tbonfort@terriscope.fr @tbonfort


Creative Commons License This work is licensed under a Creative Commons Attribution 4.0 International License.