Script to automatically set the elevation in the editor

[Bulletpoint]

7 hours ago, Butschi said:

I decided to have a small intermediate release. It is also officially version 1.0.0.

https://github.com/DerButschi/CMAutoEditor/releases/tag/1.0.0

The main change is that I added a tool to convert official German digital elevation models into a CMAutoEditor compatible format. You can find it in the folder data_conversion. It has several command line arguments:

• --dgm-dir (required): path to the directory in which you stored the dgm*.xyz.gz files
• --bounding-box (required): x_min, y_min, x_max, y_max coordinates of the box you want to extract. The coordinates have to be in the coordinate system the data points are stored in (e.g. for NRW EPSG:25832, i.e. UTM zone 32N). There are lots of tools in the internet with which you can convert from latitude/longitude to UTM, e.g. here.
• --contour (optional): set the distance between contour levels. 5 means you will have a contour line every 5m in elevation. You will get a two output files, one with a value for each square and another one with elevation value only for contour lines.
• --output-name (optional): a common name for output files. You will get several output files with a name that starts with this string.
• --water-level-correction (optional): x_low, y_low, x_high, y_high Sets the low and high point for water level correction (see below for an explanation).

There will be four output files: output-name.csv, output-name_contour.csv, output-name.png (a heat map of the elevation values. units are CM squares.) and output-name_contour.png, which has contour lines of the data you extracted.

The actual data is availble separately for each German federal state, e.g.

North Rhine-Westphalia

Hessen (Fulda, etc. ...)

Brandenburg & Berlin

... I just realized that only the data from North Rhine-Westphalia is in .gz files, the others seem to zipped. I'll make that work in the next update. Oh well.

 

Water level correction:

This is an odd one. While you are all well aware that water usually flows downhill, rivers in CM are actually lakes, i.e. every square belonging to the same water body has the same water level. In reality, this is of course not the case. But on the map I am currently experimenting with, the difference in elevation along the river is roughly 6m. This leads to the river looking fine at one side:

mg]https://i.imgur.com/DNrYDfO.png[/img]

And being in a deep ravine on the other side:

Water level correction is still a bit experimental but it should in principle get rid of this. You provide the lowest point of the river on the map (UTM coordinates) and the highest point and the tool will calculate a calculate a plain and lower the higher side (and all the other terrain accordingly) to the level of the lower side.

Contour Lines:

@Bulletpoint

I added this option because there were concerns about performance and stair-step effects. And of course it takes much less time to only set contour lines. I tested versions and could not observe any difference in performance. And for stair-step effects...

Every point set:

5m contours:

10m

Doesn't look precisely stair-step effect free to me... Am I doing something wrong?

Ah, one final thing: I changed the license under which the code is published. I can't really see how this affects you but I don't want anyone thinking I'm sneaking an evil ninja license in . You can do whatever you want with the code, you can even sell maps you made with it (as long as BF is ok with it, that is). I only speak broken legalese but I thought the new license might be a bit more fitting for community made stuff: the only restriction is, if you modify the code and make something new with it, you have to make it open source just like the original one.

Right, have fun making maps!

 

.

Thanks a lot for sharing.

I'm surprised it makes those stair step effects when given contour lines. I'm guessing it has to do with how exactly your program inputs the elevation data.

It looks to me like maybe it still inputs a control point for every square of the map, just now keeping bands of control points the same altitude. To properly blend the terrain, the engine needs free space between the control points.

[Bulletpoint]

This tool might be interesting for @theforgerwhen doing his enormous historically accurate campaigns.

[RockinHarry]

10 hours ago, Butschi said:

I decided to have a small intermediate release. It is also officially version 1.0.0.

https://github.com/DerButschi/CMAutoEditor/releases/tag/1.0.0

The main change is that I added a tool to convert official German digital elevation models into a CMAutoEditor compatible format. You can find it in the folder data_conversion. It has several command line arguments:

• --dgm-dir (required): path to the directory in which you stored the dgm*.xyz.gz files
• --bounding-box (required): x_min, y_min, x_max, y_max coordinates of the box you want to extract. The coordinates have to be in the coordinate system the data points are stored in (e.g. for NRW EPSG:25832, i.e. UTM zone 32N). There are lots of tools in the internet with which you can convert from latitude/longitude to UTM, e.g. here.
• --contour (optional): set the distance between contour levels. 5 means you will have a contour line every 5m in elevation. You will get a two output files, one with a value for each square and another one with elevation value only for contour lines.
• --output-name (optional): a common name for output files. You will get several output files with a name that starts with this string.
• --water-level-correction (optional): x_low, y_low, x_high, y_high Sets the low and high point for water level correction (see below for an explanation).

There will be four output files: output-name.csv, output-name_contour.csv, output-name.png (a heat map of the elevation values. units are CM squares.) and output-name_contour.png, which has contour lines of the data you extracted.

The actual data is availble separately for each German federal state, e.g.

North Rhine-Westphalia

Hessen (Fulda, etc. ...)

Brandenburg & Berlin

... I just realized that only the data from North Rhine-Westphalia is in .gz files, the others seem to zipped. I'll make that work in the next update. Oh well.

 

Water level correction:

This is an odd one. While you are all well aware that water usually flows downhill, rivers in CM are actually lakes, i.e. every square belonging to the same water body has the same water level. In reality, this is of course not the case. But on the map I am currently experimenting with, the difference in elevation along the river is roughly 6m. This leads to the river looking fine at one side:

mg]https://i.imgur.com/DNrYDfO.png[/img]

And being in a deep ravine on the other side:

Water level correction is still a bit experimental but it should in principle get rid of this. You provide the lowest point of the river on the map (UTM coordinates) and the highest point and the tool will calculate a calculate a plain and lower the higher side (and all the other terrain accordingly) to the level of the lower side.

Contour Lines:

@Bulletpoint

I added this option because there were concerns about performance and stair-step effects. And of course it takes much less time to only set contour lines. I tested versions and could not observe any difference in performance. And for stair-step effects...

Every point set:

5m contours:

10m

Doesn't look precisely stair-step effect free to me... Am I doing something wrong?

Ah, one final thing: I changed the license under which the code is published. I can't really see how this affects you but I don't want anyone thinking I'm sneaking an evil ninja license in . You can do whatever you want with the code, you can even sell maps you made with it (as long as BF is ok with it, that is). I only speak broken legalese but I thought the new license might be a bit more fitting for community made stuff: the only restriction is, if you modify the code and make something new with it, you have to make it open source just like the original one.

Right, have fun making maps!

 

.

Great work!

Re water level I´d just include the ground level one and leave the upstream height changes to be dealt with by map makers. It´s the games hardcoded nuisance anyway and I think it can just be dealt with by means of modding. I.e mud or marsh terrain tiles looking like water or similar ideas. Off course it doesn´t have similar properties like real CM water, but that can be dealt with in various ways. And the smaller the stream/river is, the easier the adaption. So something between small creeks (which we do have in game already) and canal size maybe.

Re terrain steps and contour lines. Height differences between adjoining action spots of 5m or more should be avoided.  These create "cliff" (impassable) terrain. So i.e when 10m contour lines are beeing used, make sure you´ve at least 2-3 AS between different height lines so the games auto mesh height adaption can avoid placing 5m ones between them. Placing roads or its far lesser visible brother "foot path" can bits of smooth out things, as well as canceling the "cliff" status. I do this a lot for my own hand drawn maps if I think "what..that 5m intervall is impassable? Ridiculous and unrealistic"...again. So properly building Ardennes or hurtgen forest like hilly terrain is doable AND playable (even for AIP if doing certain things). Lots of work though, but if one wants sort of terrain like that and not having cliffs placed all over the map, worth the efforts IMO.

Examples:

Editor view. Lots of 5m+ steps between AS´s in here. "Smoothed" with foot paths.

looks like (no cliffs)

and canopy on

Edited October 26, 2022 by RockinHarry

[Butschi]

10 hours ago, Bulletpoint said:

Thanks a lot for sharing.

I'm surprised it makes those stair step effects when given contour lines. I'm guessing it has to do with how exactly your program inputs the elevation data.

It looks to me like maybe it still inputs a control point for every square of the map, just now keeping bands of control points the same altitude. To properly blend the terrain, the engine needs free space between the control points.

My program does nothing fancy it only automates clicking. I did a bit of testing and I've come to the conclusion that this is just how the editor works and it seems it isn't really made for contour lines as input.

Just do the following experiment. Draw vertical elevation lines in the editor like illustrated below. You will always have something like an s-curve (some spline I guess) between two parallel lines. If you have three parallel lines you will just have two s-curves and you will always have a plateau in between. If you space out both lines enough the plateau will be less pronounced but it will still be there.

This will look good if you just have one contour line at the bottom of a hill and one at the top. The way the manual describes it, this may be the intended use. But it does not work for (realistic) contour lines.

I only tested it in CMCW. I don't think the different flavours have different behaviours, or do they?

[Butschi]

11 hours ago, RockinHarry said:

Re water level I´d just include the ground level one and leave the upstream height changes to be dealt with by map makers. It´s the games hardcoded nuisance anyway and I think it can just be dealt with by means of modding. I.e mud or marsh terrain tiles looking like water or similar ideas. Off course it doesn´t have similar properties like real CM water, but that can be dealt with in various ways. And the smaller the stream/river is, the easier the adaption. So something between small creeks (which we do have in game already) and canal size maybe.

It is optional, you don't have to use it. It did work fine for the example I showed above but there are certainly cases where it will fail miserably - e.g. when the height profile of the river is very nonlinear. Of course more than one such river will also not work.

[Bulletpoint]

4 hours ago, Butschi said:

My program does nothing fancy it only automates clicking. I did a bit of testing and I've come to the conclusion that this is just how the editor works and it seems it isn't really made for contour lines as input.

Just do the following experiment. Draw vertical elevation lines in the editor like illustrated below. You will always have something like an s-curve (some spline I guess) between two parallel lines. If you have three parallel lines you will just have two s-curves and you will always have a plateau in between. If you space out both lines enough the plateau will be less pronounced but it will still be there.

This will look good if you just have one contour line at the bottom of a hill and one at the top. The way the manual describes it, this may be the intended use. But it does not work for (realistic) contour lines.

I only tested it in CMCW. I don't think the different flavours have different behaviours, or do they?

I know your program only automates clicking, but I'd like to see a screenshot of the map in the editor where you can see where the actual control points were placed ?

My experience with the editor is that to get a smooth hill, you need contour lines with a certain quite short distance between them, and no control points in between.

[RockinHarry]

1 hour ago, Butschi said:

It is optional, you don't have to use it. It did work fine for the example I showed above but there are certainly cases where it will fail miserably - e.g. when the height profile of the river is very nonlinear. Of course more than one such river will also not work.

Optional is always a good thing. Guess it saves lots of time if i.e just height data and countours are needed for conversion. For rivers I think it´s quite perfect for flat areas like the Netherlands etc.

[Butschi]

New release 1.0.1:

https://github.com/DerButschi/CMAutoEditor/releases/tag/1.0.1

Change log:

• Added "stride" command line argument for dgm2cm.py. If set, the output will contain only every stride-th data point.
• removed restriction on file names in dgm2cm.py. The script will now read in every file in the given directory. This should enable to read dgm-files from e.g. Hessen and Berlin/Brandenburg (if unzip and with non xyz files removed).

 

[Butschi]

43 minutes ago, Bulletpoint said:

I know your program only automates clicking, but I'd like to see a screenshot of the map in the editor where you can see where the actual control points were placed ?

My experience with the editor is that to get a smooth hill, you need contour lines with a certain quite short distance between them, and no control points in between.

Well, some of the lines are rather close, of course. This is for 10m contour lines.

Though, if you look at the previous picture of 10m contour lines, you will see that the effect is also present in map regions where the lines are much farther apart.

I have now included an option that outputs only every n-th point. This here is every 10th point of 20m contour lines. (different map, not the same as in the editor above!) If you look close enough, you can still see the steps:

editor:

I can't really put that much more space between them...

 

Edited October 26, 2022 by Butschi

[Bulletpoint]

9 hours ago, Butschi said:

Well, some of the lines are rather close, of course. This is for 10m contour lines.

Though, if you look at the previous picture of 10m contour lines, you will see that the effect is also present in map regions where the lines are much farther apart.

I have now included an option that outputs only every n-th point. This here is every 10th point of 20m contour lines. (different map, not the same as in the editor above!) If you look close enough, you can still see the steps:

editor:

I can't really put that much more space between them...

 

Thanks for trying out different things. This version is the smoothest and least stair-steppy you'll be able to get in Combat Mission, as far as I'm aware.

If it's true that there's no performance hit to having many control points, then I think the first version with the continual contour lines is better than the sparse one. Easier to get an overview of the contours.

[Butschi]

On 10/26/2022 at 3:05 AM, Lucky_Strike said:

Map pic looks great, does it look familiar to you? Does it look like home?

More the pic I posted in the Christmas Challenge. That one really brings back memories.

[Lucky_Strike]

2 hours ago, Butschi said:

More the pic I posted in the Christmas Challenge. That one really brings back memories.

Ah yes, I remember. This is the kind of thing that makes scenarios much more playable IMHO, having a terrain that is natural and reminiscent of something remembered or something seen. Basing scenarios on real topographical information makes a massive difference to this, and having something like your script to do a lot of the heavy lifting is a great advantage.

[RockinHarry]

Just found the 3D data required for my projects here, Download tab https://www.tim-online.nrw.de/tim-online2/

Brings me one important step closer to trying out the Python scripts very soon. What Python version would be required for Win10 and not providing any hassles for my system? I once had some on my old WinXP computer, but can neither recall what it´s exactly nor what I needed it for.

Edited November 2, 2022 by RockinHarry

[Butschi]

10 hours ago, RockinHarry said:

What Python version would be required for Win10 and not providing any hassles for my system?

I'd recommend to use either anaconda3 or miniconda3. The latter is the light weight version of the former. If you like a GUI instead of just a shell, maybe take the former. On the other hand I show how to install miniconda3 in my tutorial video. One neat thing about conda in general is that you can have different environments. So you can have different packages for different projects, so you won't destroy your working installation for one project by installing new packages for a different project.

Out of curiosity: What kind of map do you use? I thought about how historical scenarios don't benefit too much from present day OpenStreetMap, so the next update will have input in geojson format, which for the uninitiated just means that you can use the "map editor" of your choice (e.g. JOSM or QGIS). With that you can just have a (geo-aware (easy to do, you just need to make an extra file which basically says where the map area is)) map image as background and paint polygons and lines on it and mark them as, e.g. forest. Should still be way easier than doing it in the CM-editor (plus you can have randomized trees in forests and all that).

Edited November 3, 2022 by Butschi

[RockinHarry]

26 minutes ago, Butschi said:

I'd recommend to use either anaconda3 or miniconda3. The latter is the light weight version of the former. If you like a GUI instead of just a shell, maybe take the former. On the other hand I show how to install miniconda3 in my tutorial video. One neat thing about conda in general is that you can have different environments. So you can have different packages for different projects, so you won't destroy your working installation for one project by installing new packages for a different project.

Out of curiosity: What kind of map do you use? I thought about how historical scenarios don't benefit too much from present day OpenStreetMap, so the next update will have input in geojson format, which for the uninitiated just means that you can use the "map editor" of your choice (e.g. JOSM or QGIS). With that you can just have a (geo-aware (easy to do, you just need to make an extra file which basically says where the map area is)) map image as background and paint polygons and lines on it and mark them as, e.g. forest. Should still be way easier than doing it in the CM-editor (plus you can have randomized trees in forests and all that).

Thanks! Will try using the mini version then. Also sounds somewhat familiar to me. Maybe it was what I´d used couple years ago. Also can refer to your video then.

For first try I selected a square 4x4km area around town of Vossenack in hurtgen forest area. But it´s 16 data packages and then I´ll try getting them fed to your scripts and CM editor.

For given area I´ve both historical maps as well as traveling the area several times. So I know fairly well what´s 1944 and what post war. Forested areas changed considerably since then, as is the villages that expanded to cleared areas around them. So IMO should be well prepared to tackle this one. For others depend on available source materials. I´ve most of what interests me for 1944/45 western germany and some other areas. But will surely be interesting to try out what you´ve in mind ATM. But can´t really judge yet. Cheers

[Butschi]

I had to keep radio silence for a while to be able to concentrate - my girlfriend noted recently that by now I have coded enough stuff for a decent bachelor thesis at least. Well, I did rewrite parts of the code at least three times and some of the algorithms were intriguingly finicky, at least if you have wood shavings for a brain as I sometimes seem to have.

There is still metric tons of things to do and to tweak. But sometimes I have to remind myself why this is fun, so I had to indulge in some screenshots and I guess it is only fair to share a few (shamelessly cherry-picked, mind you!).

Enjoy!

[Butschi]

Maybe it's interesting to have the acutal map for this, too. The sharp eyed among you can probably spy lots of things where conversion from OpenStreetMap is not yet working as it should.

Location in OpenStreetMap

[RockinHarry]

1 hour ago, Butschi said:

Maybe it's interesting to have the acutal map for this, too. The sharp eyed among you can probably spy lots of things where conversion from OpenStreetMap is not yet working as it should.

Location in OpenStreetMap

I very much like what I see! Hope you and your GF are still fine together.

Ah... near Overath. Wasn´t there some fighting when the US pushed north to divide the Ruhrkessel? Some german paras where in the area as well IIRC. Got to reread Mues book again.

I think your script is already doing more than one could´ve dreamed of not long ago. Kudos!

[Artkin]

I'm a little late giving this a try but HOLY ****!!!!!!!!!

I followed the video tutorial and got it to work

[Artkin]

Where do I go about getting CSV files? I see OSM has an export feature, I tried that, and then used a conversion website to convert the .osm file into .csv files. But I did not see any elevation files included.

[Lucky_Strike]

18 hours ago, Butschi said:

There is still metric tons of things to do and to tweak. But sometimes I have to remind myself why this is fun, so I had to indulge in some screenshots and I guess it is only fair to share a few (shamelessly cherry-picked, mind you!).

Really appreciate how this is working now, especially liking the mixed woodland in your valley. Thanks for updating us.

[SchnelleMeyer]

It's really exiting to see what you have accomplished with your work so far Butschi. A new world of map making seems to be opening up for the community with this progress. The terrain and nature elements in your screenshots looks amazingly realistic. 

[Butschi]

2 hours ago, Lucky_Strike said:

Really appreciate how this is working now, especially liking the mixed woodland in your valley.

All those area features were rather simple to do. The mixture is also quite easy to configure, e.g.:

    "broadleaved_forest": {

        "tags": [

            ["landuse", "forest"],

            ["natural", "wood"]

        ],

        "required_tags": [

            ["leaf_type", "broadleaved"]

        ],

        "cm_types": [

            {"menu": "Foliage", "cat1": "Tree A", "cat2": "density 1", "weight": 1},

            {"menu": "Foliage", "cat1": "Tree A", "cat2": "density 2", "weight": 2},

            {"menu": "Foliage", "cat1": "Tree A", "cat2": "density 3", "weight": 1},

            {"menu": "Foliage", "cat1": "Tree B", "cat2": "density 1", "weight": 1},

            {"menu": "Foliage", "cat1": "Tree B", "cat2": "density 2", "weight": 2},

            {"menu": "Foliage", "cat1": "Tree B", "cat2": "density 3", "weight": 1},

            {"menu": "Foliage", "cat1": "Tree C", "cat2": "density 1", "weight": 1},

            {"menu": "Foliage", "cat1": "Tree C", "cat2": "density 2", "weight": 2},

            {"menu": "Foliage", "cat1": "Tree C", "cat2": "density 3", "weight": 1},

            {"menu": "Foliage", "cat1": "Tree D", "cat2": "density 1", "weight": 1},

            {"menu": "Foliage", "cat1": "Tree D", "cat2": "density 2", "weight": 2},

            {"menu": "Foliage", "cat1": "Tree D", "cat2": "density 3", "weight": 1},

            {"menu": "Foliage", "cat1": "Tree F", "cat2": "density 1", "weight": 1},

            {"menu": "Foliage", "cat1": "Tree F", "cat2": "density 2", "weight": 2},

            {"menu": "Foliage", "cat1": "Tree F", "cat2": "density 3", "weight": 1},

            {"menu": "Foliage", "cat1": "Tree G", "cat2": "density 1", "weight": 1},

            {"menu": "Foliage", "cat1": "Tree G", "cat2": "density 2", "weight": 2},

            {"menu": "Foliage", "cat1": "Tree G", "cat2": "density 3", "weight": 1}

        ],

        "process": [

            "type_random_individual"

        ],

        "priority": 1

    },

The script will randonly choose one of the "cm_types" for each square where the probability is determined by the weight.

1 hour ago, SchnelleMeyer said:

The terrain and nature elements in your screenshots looks amazingly realistic. 

They do, but:

20 hours ago, Butschi said:

shamelessly cherry-picked

Buildings in particular look somewhat sterile without hedges, gardens, etc. Though, of course, there is only so much I extract from a map. Hedges are possible but are often not in OpenStreetMap. So, scenario designer won't be unemployed all of a sudden. It is easy to add hedges to OSM data, though, it's just drawing a line which is probably faster than doing it in the CM editor. This is also the way to do historical maps or maps from scratch, I think. I'll do a little tutorial on that... at some point. Oh well.

EDIT: Seems like the code block is not displayed correctly. Well, I how the general structure is visible...

 

 

Edited November 15, 2022 by Butschi

[Butschi]

21 hours ago, RockinHarry said:

Ah... near Overath. Wasn´t there some fighting when the US pushed north to divide the Ruhrkessel? Some german paras where in the area as well IIRC. Got to reread Mues book again.

Interesting, I thought Overath didn't see much more of the war than the train station being bombed.

[Butschi]

Unfortunately OSM doesn't have elevation data. There are a few resources in this post:

There are a lot of other sources and I can't possibly write conversion scripts for all of them, I'm afraid. I plan to write something for ALOS World 3D - 30m (AW3D30), it's free sattelite 3D data with a resolution of 30m. Not awesome but available for most of the globe. Generally, though, you should go for digital terrain models from airborne LIDAR scans that have been corrected for terrain features like buildings, etc.