MapServer Project Status Report

Project Status Report

FOSS4G-Europe 2015 - Como - 2015-07-15

Stephan Meißl stephan.meissl@eox.at @Schpidi

Thomas Bonfort tbonfort@terriscope.fr @tbonfort

This work is licensed under a Creative Commons Attribution 4.0 International License.

What is MapServer?

Open Source platform for publishing spatial data to the Web
MIT-style license
Runs on all major platforms (Windows, Linux, Mac OS X)
OSGeo project
mapserver.org

MapServer History

1994: Steve Lime implements Web-based Arc/Info AML generator
1994-95?: Frank Warmerdam releases Shapelib
1997: v1.0 (limited sharing)
1998: v2.0 via Fornet website
1999: v3.0 via UMN MapServer website
2001: PostGIS support by Paul Ramsey in v3.6
2002: GRASS Users Conference - Trento
2003-05: Yearly MapServer Users Meeting
2006: Foundation of OSGeo with MapServer as founding project
2006+: FOSS4G Conferences
2009+: C Tribe Code Sprints
2012: Addition of TinyOWS & MapCache ⇒ MapServer Suite (v6.2)
2013: v6.4
2014: v7.0beta & 20th anniversary
2015: v7.0 (beta2 released 07/2015)

Statistics

~150k Lines Of Code
~12k commits
~100 contributors
Mailing lists
- -users ~1800
- -dev ~400
International PSC

14 members
NA: 9 (US: 5, CA: 4)
EU: 5 (FR: 2, FI: 1, AT: 1, HU: 1)

113 RFCs

MapServer 7.0

Main new features

UTFGrids
WFS 2.0
Heatmaps/Density Maps
Javascript/V8 StyleItem/Geomtransform
Unification of attribute/geometry filtering
Layer Level Compositing / Blending

Others
- Text rendering re-factoring
- GD Removal
- Removed BITMAP labels
- Layer Level Encoding
- WCS 2.0

UTFGrids

Specification by MapBox for vector tiles
Implemented in a "Google Summer of Code" project
Interactivity important (rollover, events)
Supported by OpenLayers

WFS 2.0

Download Services for INSPIRE

Javascript (V8) StyleItem/GeomTransfom

Integration of the V8 JavaScript engine
Style geometries/features programatically rather than through class expressions
"Experimental mode": backward compatibility not guaranteed


LAYER
    ...
    STYLEITEM "javascript:///path/to/my/file.js"
    CLASS # empty class required
    END
END


//Make symbol size 14 or 7
  var size = shape.attributes.NAME.length > 10 ? 14:7;

  var style = "STYLE SIZE " + size;
  style += " SYMBOL 'circle'";
  style += " COLOR 255 0 0 END";

  //Return style to MapServer
  style;

Unification of attribute/geometry filtering

Currently filtering done by MapServer
PostGIS, Oracle, OGR support native spatial filtering e.g.


where st_intersects(the_geom, st_geomfromtext('wkt...'))

Significant performance gain particularly in WFS queries
Simpler to use
Was blocking release of version 7.0

WCS 2.0

Implements available extensions
Passes all available CITE tests
Official OGC WCS 2.0 Reference Implementation

HeatMaps

Vector to raster processing pipeline
Interpolation and rendering parameters

Automatic scaling
Interpolation in color spaces RGB/HSL
Weighting per attribute/expression

Supports tile mode

Interpolation in HSL color space

Interpolation in RGB color space

Pure density

Setting a fixed scale

Layer Blending

Layer Filters

Text rendering re-factoring

Before naive implementation duplicated for each back-end
Integration of Harfbuzz library
Supports complex text layout e.g. for Arabic text

Unprocessed characters
With bi-directional processing
With glyph shaping rendering letters according to context

Barely visible to the end user, but

Exact label placement (alignment/centering/letter spacing of text)
Possibility of using one font per language

Improving performance (notice the log scales)

Malayalam Example

Tibéto-Birman Example

GD Removal

Added back in 1995
Too high maintenance costs despite its usefulness in certain niches
Automatic fall-back to AGG + 8bit

Removed BITMAP labels

Truetype font included by default now
Supports many glyphs without external dependencies
A few use cases are not supported
Automatic fall-back in place

Layer Level Encoding

Solves encoding handling problems e.g. data sources in multiple encodings in a single mapfile
Data sources internally converted to UTF8
The mapfile has to be encoded in UTF8 now
All output generated by MapServer in UTF8

MapCache 1.4

Tiling Library with Apache, FastCGI, Nginx and Node frontends
Versatile
- Multiple Cache Backends
- Multiple Protocols
- Tile Management Features: seeding, recompression, interpolation
Native code: Fast and Efficient

Standard Tile Addressing

TMS
WMTS (RestFull + KVP)
VirtualEarth QuadKeys
MapGuide
Google x,y,z

WMS

WMS-Tiled
Responds to untiled requests by assembling tiles
Can mirror a full WMS service: layer-by-layer, layer lists
Forwards GetFeatureInfo requests

(OGC) Proxy

Intercepts requests that can be served from cached tiles
Forwards other requests to other upstream server(s)

Miscellaneous

Caching HTTP Headers
"Auto" Expiry of old tiles
Error Reporting
- Error Message
- Empty Image
- Status Code
Meta-Tiling
Watermarking
Max-Cached-Zoom: Upsample lower level tiles

Image Operations

Image data can be recompressed/optimized before being stored into a tile
- Avoid double JPEG compression, i.e. request PNG from source WMS
- Be agressive on compression when storing a tile (e.g. quantized PNG with max compression)
- CPU time for dynamic recompressions can be lowered, at the cost of more bandwidth

Image Formats

PNG
- Compression level
- Quantization (color reduction to 8bit palette)
JPEG
- Compression level
- RGB/YCbCr
Mixed (JPEG + PNG)
- Use for raster coverages on limited areas
- PNG for tiles with transparency
- JPEG otherwise

Handling Empty/Uniform Tiles

Strategy dependent on the cache backend
- Symbolic linking
- Reconstruct Paletted 1bit PNG from uniform color
- For fully seeded caches, treat absent tile as fully transparent

Grids

Multiple Grids per Tileset (e.g. WGS84 + Mercator)
Handles Grid Aliasing
- EPSG:3857/EPSG:900913
- Non Standard Codes
Grid Subsets
- min/max resolution
- Restricted extent (with tolerance!)

Caches

Backend able to store tile data for given x,y,z
Extensible: Exists(), Get(), Put(), Delete()
Some backend specific hacks: symlinking, blank detection

Disk Cache

Mainly for dev, testing, or small tilesets
Pros
- Simple, just provide a base directory
- Fast
- Supports blank tile detection
- (Re-)Use custom layouts: TileCache, ArcGis, Worldwind, Template
Cons
- Difficult to manage large number of files
- Hits File System Limits
- Wasted space due to File Sytem block size

Sqlite Cache

Stores tile data as blobs in an sqlite file (/path/to/{grid}/{tileset}/{z}.db)
Can be split into multiple db files (per dimension, grid, tileset. Configurable max number of tiles per db)
Pros
- Single File
- Support any schema (MBtiles, custom...)
- Efficient Space (blank tile detection possible)
Cons
- Some tweaking needed for very large caches (more than 1 Tb)
- Concurrent Insertions are slow

MemCache

Ideal for Temporary Data: Forecasts, Sensors, ...
Pros
- Easily distributed
- Automatic pruning
Cons
- Limited Storage

REST Caches

REST Back-end

Support GET/PUT/DELETE for storing tiles
Authentication/authorization plug-ins for S3, Azure, and Google Cloud Storage
Costs highly depend on the use case (number of applications, bandwidth)

(Geo)TIFF Cache

Specialized storage for satellite imagery basemaps
JPEG tiles are read directly from a constrained (Geo)TIFF file hierarchy
Configurable number of tiles per TIFF file
Templated access to file name
Pros
- Efficient on disk usage (TIFF container)
- TIFF files can be shared with other applications
Cons
- Limited to JPEG data (no transparency)
- No Concurrent writing to a single TIFF file
- Updates/deletion of individual tiles does not reclaim storage space
Use-Case

Misc Caches

Riak (distributed, extensible, fault tolerant)
Berkeley DB
Couchbase (soon)
Redis (soon)

Composing Caches

Composite Caches Select different caches based on rules (zoom level, grid)
Fallback Caches
Multitier Caches mix caches with different performance/storage characteristics. If a tile is not found in the fast/small/expensive/transient cache, it will be looked for in the slow/large/cheap/permanent cache and moved into the first one for subsequent accesses.

Seeder: Pre-generating tiles

Multi thread/process seeding
"Drill-Down" mode to take advantage of caching on source WMS
Seed particular subset
- Given Dimension
- Regenerate old tiles
- Restricted min/max zoom and/or extent
- Restrict to geometry (e.g. seed to level 18 all areas where population density is over a given value)
Pruning mode