Saturday 26 December 2015

Mapping a real time snow cover

Norway is a country with huge climate variations between seasons, lowlands and mountains, coastal areas and inland, and between north and south. In the recent years, we have also seen more extreme weather and greater fluctuations in temperature and snow cover. At the same moment of time there might be meters of snow in the mountains, while the trees are blooming by the fjord a few kilometers away. Outdoor activities are popular among citizens and tourists alike, and in Norway you can do typical summer and winter activities all year around. The goal of this blog post is to create a near real time map of the snow cover, where the snow blends into the landscape.

The snow map of Jountunheimen we're going to create.

As shown on this blog, I like to create maps and 3D visualisations using a wide range of open source tools, combined with various programming techniques. This snow mapping experiment is no exception. Open Source allows me to mix and match the tools and techniques I need, and doing this programmatically has the added benefit of automation. When the full map making process is defined, we can easily run it for various areas and terrains. It was an important goal of this project to keep the snow cover up­-to-­date on a daily basis.

Cross­country skiing in Jotunheimen national park. Photo: Bjørn Sandvik
Geospatial datasets can be very large, and we often have to divide the data into chunks to be able to run calculations. I have divided mainland Norway into 218 cells, each covering an area of 50 x 50 km. With datasets at 10 meters resolution, all tools presented below were capable of processing the data.

The Norwegian Mapping Authority released its topographic datasets in 2013. This included an elevation model of mainland Norway at 10 m resolution. The Norwegian Water Resources and Energy Directorate (NVE) is monitoring the country's water resources, and they provide data of snow gain, snow melt and snow depth. As my map is aimed towards cross­-country skiers, I’m using a dataset showing areas with a skiing surface, based on interpolated weather observations.

Elevation data
The 10 m digital elevation model (DEM) from the Norwegian Mapping Authority was used to blend the snow into the landscape. A few GDAL commands allowed me to create a 50 x 50 km DEM from the nation­wide DEM, and to translate it into the various formats supported by the tools in use.

The 10 m elevation model contains too much detail to represent a snow cover. I could downsample the dataset, but that would blur out important terrain features like mountain ridges and valley edges. Terrain Sculptor came to the rescue. It is a tool by Bernhard Jenny and Anna Leonowicz to generalize terrain models for relief shading. It removes unnecessary and distracting terrain details, and better resembles a smooth snow covered landscape. Unfortunately, Terrain Sculptor can only be used with a graphical interface, and not as a command line tool, so it can not be used in an automatic process. I used it to create a generalised terrain model for the 50 x 50 km area of Jotunheimen and Tyin.

The generalised elevation model was used to create a shaded relief. We can easily create a grayscale hillshade with GDAL, but the snow required some tints to look better. I created a continuous colour scale from blue shadows to white with a touch of yellow, and turned the grayscale hillshade into a tinted hillshade. For areas without snow, I used the same elevation data to create a color relief progressing from green for lower elevations up through yellows/browns, and on to grays and white at the highest elevations.

Finally, the elevation model was used to calculate steep terrain (slope) where there will be less snow. The steep terrain will be subtracted from the snow surface to improve the view in alpine areas. This was achieved in a series of steps: A slope surface was calculated from the DEM, and then areas steeper than 50 degrees were extracted and vectorised. The process of vectorising raster data is described below.

Snow cover
The skiing surface dataset from the Norwegian Water Resources and Energy Directorate (NVE) is available from a public Web Map Service (WMS) with a 1 km resolution. The dataset is updated 8 times a day. The original WMS image includes 4 colours: Blue represents skiing surface with dry snow, violet is surface with wet snow, yellow is areas with little snow, and green is areas without snow. As the map is targeted towards cross country skiers, the dry (blue) and wet (violet) snow are combined into one layer. The areas with little snow (yellow) are not included in this map, but could be used to create a more gradual shift between snow cover and bare ground.

The original WMS image of skiing conditons.

Wet at dry skiing conditions combined. Each square is 1 x 1 km. 

To avoid the pixelated appearance of the snow surface, the image is vectorised into smooth scalable snow polygons. This was achieved using Potrace by Peter Selinger, an open source tool for tracing bitmaps. Potrace transforms the the blocky WMS image into smooth vectors. It supports GeoJSON output, which makes it easier to use in geographic applications. A problem with this technique is that the smooth polygons seem more accurate than they are. There are other, and probably better, ways to interpolate the data, as following height contours in the area.

Vectorised image from Potrace.

With all the map layers in place, Mapnik was used to generate the map shown below. Mapnik is an open source toolkit to generate beautiful maps from different vector and raster sources. The order and styles are expressed in an XML document, and then applied to the data. First, colour relief, lakes, snow polygons, steep terrain, glacier polygons and road lines are added in turn. Then all the layers are blended with the hillshade, before adding buildings and a thin lake outline as a final touch.

Jotunheimen and Tyin covered with a snow cover obtained from the Norwegian Water Resources and Energy Directorate on. The map image is applied to a 3D model using elevation data from the Norwegian Mapping Authority.


Friday 6 November 2015

Mapping grid-based statistics using OpenLayers, Three.js and D3.js

I've just finalised two tutorials on mapping grid-based statistics. The tutorials are in Norwegian, but the source code is available in English, and should be easy to follow.

Tutorial 1: "Grid-based population data"

Mapping grid-based population data of Oslo, Norway, using OpenLayers 3 and D3.js. The user can select an area to see the number of inhabitants.


Tutorial | Demo | Source

Tutorial 2: "Grid-based statistics in 3D"

3D visualisation of grid-based population data of Oslo, Norway, using Three.js and D3.js.


TutorialDemo | Source

The basemap is from the Norwegian Mapping Authority, and the grid-based population data from Statistics Norway.

The tutorials are provided by Project Innovation from GeoForum, a Norwegian non-profit industry organization for individuals and companies/agencies working in mapping, surveying and geographic information.

Please notify me if you use these techniques in your own projects! 

Wednesday 14 October 2015

Mapping the Arctic sea ice

There as been a lot of attention about the diminishing Arctic sea ice in recent years. We’re often exposed to images showing the extremes. My goal was to show the changing sea ice month by month, and even day by day. Sea ice is a critical component of our planet because of its influence on the global climate.

The sea ice cover in the Arctic in September 2015. The black line is the median extent (1981-2010) for this month. See interactive version.


The graph shows the change in sea ice cover for September since 1979, in percent from the mean extent (1981-2010). September 2015 had the fourth lowest extent in the satellite record. See interactive version.

The National Snow and Ice Data Center (NSIDC) provides two great datasets collected by satellite of the Arctic Sea Ice:

Sea Ice Index
You get the longest time-series using Sea Ice Index, which dates back to November 1978. The dataset is available at 25 km resolution. The images and data are produced in a consistent way making it appropriate for use when looking at long-term trends in sea ice cover.

MASIE
MASIE is a newer dataset at 4 km resolution, which dates back to 2006. Recently they also made the dataset available at 1 km resolution (back to December 2014), which is great for regional maps. Use MASIE when you want the most accurate view possible of Arctic-wide ice on a given day.

This page further explains the difference between the two datasets.

Responsive map layout adapting to various devices. Image from Am I Responsive.

Polar maps are hard to make with popular mapping tools like Mapbox and CartoDB. I love these tools, and I use them a lot when the Mercator projection makes sense. But Mercator is a terrible match for polar data. Not only are the areas towards the poles greatly distorted, the world above 85°N or below 85°S simply don’t exist! While waiting for better projection support in these tools, you can try these hacks: Mapbox example, CartoDB example.

We should always pick the right tools for our job. For this project I ended with a geohipster stack of D3.js, GDAL and Node.js on Digital Ocean droplet. There is also a touch of CartoDB to store some tabular ice extent data, but no geodata this time.

D3 has great support for different map projections. I decided to use the same polar stereographic projection as the original sea ice datasets. This made it easier to keep the original resolution of the sea ice datasets. The only thing I did was to rotate the map to get Norway and Europe in the center - sorry to my North American readers ;-)

Sea ice maxiumum (March) and minimum (September) in 2015. See interactive version


There is a significant yearly cycle in sea ice cover, usually reaching a maximum in March, and a minimum in September. See interactive version.

Bonus information for the map geeks: The polar projection of MASIE and Sea Ice Index is not exactly the same - not only is the central meridian different (-80° vs -45°), the latitude of true scale is 60° for MASIE and 70° for Sea Ice Index. The first discrepancy is just a matter of rotating the images until they overlap. To fix the latter you have to alter one of the projections to allow them to line up.

I wanted my map to be up-to-date with the latest sea ice data available. I don’t have time to update the map every day - so I needed to do this in an automatic process. It was achieved by periodically checking for new data on the NSIDC servers, download the files and convert them to a web friendly format. The original GeoTIFF showing the sea ice extent on October 12 is 1.7 MB, while the compressed PNG shown on the map is just 12 KB, with the same 4 km resolution!

Animated GIF showing the changing sea ice extent from maxium in March 2015 to minimum in September. The animation is recorded directly from the website, showing the benefit of the highly compressed PNGs from an SSD cloud server. 

D3.js and SVG are a great match for responsive map layouts adapting to various screen sizes and resolutions. I wanted this map to be both mobile friendly and big screen friendly, and it was achieved using CSS3 flexbox and viewport units.

Please contact me if you’re interested in using this visualisation, or if you would like help on another mapping project.

Thursday 4 June 2015

Master maps

I’m going freelance over the summer, after 5 great years at the Norwegian Broadcasting Corporation (NRK). It was not an easy decision, but I have to try. I’ll tell more about my plans later. Please sign up to get notified about my services.

Some of the projects I've been working on at NRK:

The flexible mapping stack of NRK.no, allowing journalists and digital storytellers to create advanced maps in minutes. 

"Kartoteket" - our in-house mapping tool built on top of our mapping stack.

Digital storytelling using NRKs mapping stack and Mapbox. 

Digital storytelling using NRKs mapping stack and Mapbox. 

Flood maps using NRKs mapping stack and CartoDB.

Radon affected areas in Norway using NRKs mapping stack.

Our popular photo maps

Video map of the long running TV show Norge Rundt.
Tracking of "Sommerbåten" along the coast of Norway.

Other work.

Thursday 23 April 2015

Real time satellite tracking of your journeys - how does it work?

I'm back in Oslo after my 25 days ski trip across Nordryggen in Norway. It was a great journey, and I would highly recommend doing all or parts of it if you enjoy cross-country skiing. Just be prepared for shifting weather conditions.
The goal of the trip was also to test my solution for real time satellite tracking, explained in several of my previous blog posts. It worked out really well, and people were able to follow along in the comfort of their sofa.


I fastened a Spot Satellite Messenger to the top of my backpack, and left the device in tracking mode while skiing. The device sent my current position every 5 minutes, allowing me to update the map without any mobile coverage. When we arrived at a mountain hut, I pressed the OK button to set up a bed. I also programmed a button to show a snow cave, in case we wouldn't reach a hut. Luckily we didn't have to use it :-)

My map and elevation plot of the 25 days ski trip across Nordryggen. Most of the trip is above tree line, and there are only 5 road crossings in total. 

The SPOT messenger only sends my time and position, so I had to create a web service to retrieve extra information about each location. I'm using a service from the Norwegian Mapping Authority
 to retrieve the altitude, nearest place name and the terrain type. Earlier this winter, I experienced that the service did't return any altitude if I was skiing on lakes, so I'm using the Google Elevation API to avoid gaps in the elevation profile.

By knowing the time and location, I could create an automatic service to obtain more information to enrich the map. In addition to elevation and place name, I've added a weather report.  The image show Bjordalsbu, the highest lying hut on the route 1586 m, which we visited in a strong breeze. 

While skiing, I used Instagram to post photos that would instantly show on the map as well. This required mobile coverage, which is sparse in the mountains. After the trip, I synced my camera photos with my GPS track to be able to show them along the route.

Click "Bilder" in the top menu to see the photos along the route. 

A few of my photos:

Eidsbugarden in Jotunheimen. 

Iungsdalshytta in Skarveimen. 

Taumevatn in Ryfylkeheiane.

Gaukhei in Setesdalsheiane. 

End of trip - and the snow - in Ljosland. 
More photos in my Google+ album.

Friday 20 March 2015

Transferring a route from QGIS to your GPS

In my last blog post, we created a created a 500 km continuous line representing a ski route across Nordryggen in Norway. I need to transfer this route to my Garmin GPS so I can use it for navigation while skiing. How can it be done?

Cross-country skiing in Skarvheimen, Norway. Photo: Bjørn Sandvik

Open the route in QGIS, right click the layer and select "Save As...". Select "GPS eXchange Format [GPX]" as the format, and "WGS 84" as the coordinate reference system (CRS). I'm also skipping attribute creation as my line only contains coordinates.



QGIS saves the line as a GPX route. You can import this route in Garmin BaseCamp (File -> Import). Give the track a meaningful name.

My 500 km route consists of 3867 points, but most Garmin GPS units are only capable of showing 250 points per route. You can get around this limitation by converting the route into a track. Right-click the route in Basecamp and select "Create Track from Route".


Transfer the track to your GPS unit:


You should now find the track in the "Track Manager" on your GPS. Select "View Map" to see it and then "Go" if you want to navigate along it. Depending on the maps you have on your device, you can also display an elevation plot. Since the original track don't contain elevation data, the GPS will try to fetch it from your map.

My GPS is now loaded and I'm ready to go!

All data are available on GitHub.

Nordryggen on skis for 25 days - creating a route map

I’m currently doing my last preparations for a 25 days skiing trip across Nordryggen in Norway. It will of course depend on weather, snow conditions and blisters, but hopefully the conditions will be bearable. Norway has a great network of 500 cabins maintained by the Norwegian Trekking Association. The longest connected cross country skiing track I’ve found is around 500 km, - how to map it?

The map we're going to create with QGIS and CartoDB. Data from the Norwegian Mapping Authority.

Nordryggen (“the north ridge”) is a fairly new name of the 1,700 km mountain range that runs through the Scandinavian Peninsula. My plan is to ski around 500 km in the southern part of Norway, most of it above the tree line.

Cross-country skiing in Jotunheimen. Photo: Bjørn Sandvik

You can study the waymarked ski routes on UT.no, or download the data from the Norwegian Mapping Authority if you want to map it yourself.  The dataset is available as a PostGIS dump or in SOSI. SOSI is a common vector data format used to exchange geographical information in Norway, and not very well supported by various mapping applications. Luckily we have Sosicon, a great open source converter by Espen Andersen. I ran this command on the SOSI file:

./sosicon -2shp Tur\ og\ friluftsruter.sos 

In return I got 4 shapefiles for ski routes, hiking trails, bike trails and POIs.

If you open the ski routes shapefile in QGIS it won’t tell you much without a basemap. Let's use the map tile service from the Norwegian Mapping Authority. In QGIS, click on the “Add WMS/WMTS Layer” button in the left toolbar. Click the “New” button to create a connection. Add a name and copy this URL:

http://opencache.statkart.no/gatekeeper/gk/gk.open_wmts?Version=1.0.0&service=wmts&request=getcapabilities



Click “OK” and “Connect”. You will now get a long list of layers in different map projections. I recommend using “norges_grunnkart” or “topo2” (detailed) in ESPG:32633 (UTM 33).



Click on “Add” and you should see a topographic map of Norway. Now you can add the shapefile, and the ski tracks will show on top.

Waymarked ski routes shown in QGIS with my route selected between Sota Sæter and Ljosland.

I selected the track segments I plan to follow, and saved the selection in a new route shapefile. I want to have the route as a continuous line with the coordinates in order, as this will allow me to use it for navigation with my GPS unit (see my next blog post). I'm using the "Join multiple lines" plugin to get the desired result.

Installing plugins is very easy with the QGIS Plugin Manager. After installation, you'll find the "Join multiple lines" plugin in the Vector menu. 

You can use the Field calculator to calculate the length of the continuous track:


This outputs a length of 505 km.

Let's move on to CartoDB to create the route map. First, I'm uploading the zipped route shapefile to CartoDB. I've also created a table with the cabin name and positions along the route.

Create a new visualzation and add the route and cabin tables. You can add the same basemap as we used in QGIS with this URL:

http://opencache.statkart.no/gatekeeper/gk/gk.open_gmaps?layers=norges_grunnkart&zoom={z}&x={x}&y={y}



The cabins are styled differently depending on the zoom level. This is the final map:


All data is available on GitHub. Map data from the Norwegian Mapping Authority.