Script to automatically set the elevation in the editor

[Artkin]

On 9/23/2022 at 4:17 PM, Butschi said:

I wrote a little python script to set the elevation in the editor according to a digital elevation model. Those come in various resolutions, for instance, in Germany data for the whole of Northrhine Westfalia is available for free with a 1mx1m granularity. I then use PyAutoGui in order to automate setting the correcting height and clicking/scrolling in the editor. The script currently sets the height information for every square in the editor, not just contours.

An example height map:

The resulting terrain in the editor (3328mx1920m):

Vista somewhere in there.

The script is not terribly fast, making this map took about 7h, but it beats doing all the clicking yourself. The limitation, btw., is the editor, the script could go faster. The advantage is that you can do much better micro-terrain, I think, the disadvantage is that you can't just use an old topographical map and draw the contour lines. It would be possible to do contour lines but you'd need to have those in a machine readable format.

Anyway, if people here are interested, I could polish the code a bit and make it available to everyone, e.g. via github. If there is additional interest, I could imagine doing other stuff that way, like, for instance, drawing roads from Open Street Map.

Oh my god this is INSANE!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

How did I not see this ?????

Can you please make a video tutorial. I was always a hardware guy A+ certified, and github was still one of the most confusing websites on the whole wide internet for me.

This is incredible. Please make a video asap and spam it, this will blow up.

[Artkin]

@WimO might find interest in this, I know he was working on a real mountainous map

[Butschi]

10 hours ago, Artkin said:

Oh my god this is INSANE!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

How did I not see this ?????

Can you please make a video tutorial. I was always a hardware guy A+ certified, and github was still one of the most confusing websites on the whole wide internet for me.

This is incredible. Please make a video asap and spam it, this will blow up.

A video tutorial... that sounds like yet another set of tools I have to learn. I'll see what I can do...

Re: github, if you don't care about all the git-related stuff, issue tracking and what not, you can get the code with just 2 clicks: click "code"->click "Download ZIP".

[Artkin]

6 hours ago, Butschi said:

A video tutorial... that sounds like yet another set of tools I have to learn. I'll see what I can do...

Re: github, if you don't care about all the git-related stuff, issue tracking and what not, you can get the code with just 2 clicks: click "code"->click "Download ZIP".

Or just even screenshots please. I installed the code from the .zip link last night, along with python, anaconda3. I tried to install numpy and pandas but I'm unsure if I did it right?

[Butschi]

9 minutes ago, Artkin said:

Or just even screenshots please. I installed the code from the .zip link last night, along with python, anaconda3. I tried to install numpy and pandas but I'm unsure if I did it right?

Ok, I'll look into making some kind of graphical tutorial, video or otherwise. In the meantime: If you run the script, does ut complain about something?

[Artkin]

Just now, Butschi said:

Ok, I'll look into making some kind of graphical tutorial, video or otherwise. In the meantime: If you run the script, does ut complain about something?

I hadnt gotten to that point because I'm always very confused once I have to touch a code(?) and its library or database(?) I assume this is what im downloading. 

I saw that once you run the code it takes over so I didnt want to run it yet. 

I looked over the readme and understood maybe half. 

The stuff is a little too advanced for someone growing up on computers (Like me), so I doubt most ppl here will know what do either. 

That said, this has potential to be an incredibly powerful mapmaking tool, getting the hard part of it (elevation work) done automatically is incredible. 

[Butschi]

4 minutes ago, Artkin said:

I hadnt gotten to that point because I'm always very confused once I have to touch a code(?) and its library or database(?) I assume this is what im downloading. 

I saw that once you run the code it takes over so I didnt want to run it yet. 

I looked over the readme and understood maybe half. 

The stuff is a little too advanced for someone growing up on computers (Like me), so I doubt most ppl here will know what do either. 

That said, this has potential to be an incredibly powerful mapmaking tool, getting the hard part of it (elevation work) done automatically is incredible. 

Sorry, it is hard to estimate what is easy to understand for people not doing software development on a daily basis (but on linux, so I'm not to certain with Windows myself) and what isn't.

Yoy can just run the script without giving it the -i argument. It will then exit complaining that you didn't give it an input file but only after checking if the libraries are there.

[Artkin]

27 minutes ago, Butschi said:

Sorry, it is hard to estimate what is easy to understand for people not doing software development on a daily basis (but on linux, so I'm not to certain with Windows myself) and what isn't.

Yoy can just run the script without giving it the -i argument. It will then exit complaining that you didn't give it an input file but only after checking if the libraries are there.

No worries I understand. I ran the script in Anaconda3 but it gives me an error immediately about not being able to find CMAutoEditor.Py. It's looking for it in my documents, so I threw the .py file in there but that did not fix the issue. My friend might be able to help me with this, so no rush.

[Butschi]

1 minute ago, Artkin said:

No worries I understand. I ran the script in Anaconda3 but it gives me an error immediately about not being able to find CMAutoEditor.Py. It's looking for it in my documents, so I threw the .py file in there but that did not fix the issue. My friend might be able to help me with this, so no rush.

Just give it the full path to cmautoeditor.py (like e.g. python C:\User\Artkin\CMAutoEditor\cmautoeditor). Actually goung ro the directory the script is in (in the shell...) and running it there should work, given python is setup correctly.

[BornGinger]

On 9/26/2022 at 11:18 PM, Artkin said:

I was always a hardware guy A+ certified, and github was still one of the most confusing websites

This is software, though.

Edited September 28, 2022 by BornGinger

[Butschi]

By popular demand.

I'm certain I missed half the relevant points, so feel free to ask here or comment below the video.

@Artkin

 

[Artkin]

14 minutes ago, Butschi said:

By popular demand.

I'm certain I missed half the relevant points, so feel free to ask here or comment below the video.

@Artkin

 

YIPPEEEEE I was actually going to remind you the other day, this will be some groundbreaking stuff. Thanks a lot, I'll give it a try later. 

Thank you so much, over the years this post will become more and more valuable as more ppl see it. 

[Lucky_Strike]

6 hours ago, Butschi said:

By popular demand.

I'm certain I missed half the relevant points, so feel free to ask here or comment below the video.

Great job Butschi - even an eejit like me can follow your instructions! As Artkin said, this is really important and innovative work - thanks.

[RockinHarry]

Just WOW! Congrats Butschi! I´ll certainly try this one out soon. Think waiting 7 hours is nothing compared to hand editing contours on a large detail map. Can let it run overnight or if not beeing @ home. So all energy and time left for the more fun and creative things. One my prefered methods creating (non historical) maps in CM1 was letting the game generate a random map (then erasing all but contours) and then "cultivating" and settling it manually in editor. Where would be the forests, cultivated areas, settlements, communications and all that if humans started sculpting this virgin area long time ago. This then led to interesting mission design ideas to play that newly created landscape. All very fun and creative process. So with your scripts I´d likely try a similar method, creating random map contours in one the available map generators, exporting to required format then getting processed by CMAutoEditor. Fun times ahead. How about german keyboard layout (which I use)? Matter of switching a driver or changing Win10 setting? Never dealt with that before and not needed knowing it.

[Butschi]

46 minutes ago, RockinHarry said:

How about german keyboard layout (which I use)? Matter of switching a driver or changing Win10 setting? Never dealt with that before and not needed knowing it.

For me (Windows 10) it is just a matter of clicking "DEU" in the left of my screen and choosing "ENG". But indeed you may have to install some additional stuff. Einstellungen -> Zeit und Sprache -> Sprache -> Sprache hinzufügen could do the trick.

[RockinHarry]

53 minutes ago, Butschi said:

For me (Windows 10) it is just a matter of clicking "DEU" in the left of my screen and choosing "ENG". But indeed you may have to install some additional stuff. Einstellungen -> Zeit und Sprache -> Sprache -> Sprache hinzufügen could do the trick.

Thanks! I´ll dig into that when it´s the time then. Nothing overly difficult I guess.

[Butschi]

25 minutes ago, RockinHarry said:

Thanks! I´ll dig into that when it´s the time then. Nothing overly difficult I guess.

Never mind, I fixed the issue. Should work now with other keyboard layouts, too. You'll have to do pip install -r requirements.txt again, though. (after updating the code)

[SchnelleMeyer]

This is just amazing work Butschi! Looking really forward to see where your work will take the process of creating a map.

[Butschi]

So, just to give you a quick update, everything below refers to importing stuff from OpenStreetMap

Wot Is Done Already:

• Forests, randomized tree types and densities, tree types configurable, weights for types and densities configurable
• Bushes as above
• Grass as above
• Farmland as above
• water
• etc.
• (basically everything that is just a polygon in OSM can be done in this way)

Wot I'm Working On:

• roads, Roads, ROADS!!! I became a little obsessed with them. By now I'd like a word with the sadist at Battlefront who decided that doing the road tiles the way they are was a good idea. Most designers don't bother, because it is a royal PITA that you can't just combine most of the tiles, because the widths don't fit (so, you can't go from W<->E to NW<->SE directly, there has to be a tile in between. Not even the official Line Drawing Tool in the editor does it right. Oh well. After rather excessive use of graphs and shortest path finding in there and so on, I found a solution that works. I am just fixing edge cases. And then I will have to destroy it because while correct most people will not like the sometimes resulting wiggly roads instead of just diagonal ones. I guess I'm going insane!
• Once that's done though, railways and streams will be easy to implement and fences and walls should also be relatively straightforward.

Wot May Come:

• Areas (like forests, grass, etc, see above) with configurable cores/borders, such that you can have forests that are more dense on the inside or have different ground, fields with crops but just plowed ground on the edges
• (randomly) walled or fenced areas - good for having fences around fields, hedges around residential areas, ...
• Houses, but only types, I can't do 3D-view
• Flavour Objects (which you will have to fine tune yourself, no automatic stuff in 3D-view): Bus stops where there are bus stops on the map, power poles, etc.)

Well, that escalated quickly from nice little project for a weekend or two...

Edited October 21, 2022 by Butschi

[Lucky_Strike]

2 hours ago, Butschi said:

Well, that escalated quickly from nice little project for a weekend or two...

Don't we know that feeling ...

Thanks for all you're doing Butschi.

[RockinHarry]

Thanks @Butschi If you got it all working then you´ll likely trigger many map makers releasing stuff that was deemed impossible or too laborous. Me included.

[SchnelleMeyer]

Thanks for the update! This is absolutely awesome! Just look at how realistic this looks in the screenshot on this post:

Thank you so much for the time and efforts you put into this project @Butschi.

 

 

 

[Butschi]

I decided to have a small intermediate release. It is also officially version 1.0.0.

https://github.com/DerButschi/CMAutoEditor/releases/tag/1.0.0

The main change is that I added a tool to convert official German digital elevation models into a CMAutoEditor compatible format. You can find it in the folder data_conversion. It has several command line arguments:

• --dgm-dir (required): path to the directory in which you stored the dgm*.xyz.gz files
• --bounding-box (required): x_min, y_min, x_max, y_max coordinates of the box you want to extract. The coordinates have to be in the coordinate system the data points are stored in (e.g. for NRW EPSG:25832, i.e. UTM zone 32N). There are lots of tools in the internet with which you can convert from latitude/longitude to UTM, e.g. here.
• --contour (optional): set the distance between contour levels. 5 means you will have a contour line every 5m in elevation. You will get a two output files, one with a value for each square and another one with elevation value only for contour lines.
• --output-name (optional): a common name for output files. You will get several output files with a name that starts with this string.
• --water-level-correction (optional): x_low, y_low, x_high, y_high Sets the low and high point for water level correction (see below for an explanation).

There will be four output files: output-name.csv, output-name_contour.csv, output-name.png (a heat map of the elevation values. units are CM squares.) and output-name_contour.png, which has contour lines of the data you extracted.

The actual data is availble separately for each German federal state, e.g.

North Rhine-Westphalia

Hessen (Fulda, etc. ...)

Brandenburg & Berlin

... I just realized that only the data from North Rhine-Westphalia is in .gz files, the others seem to zipped. I'll make that work in the next update. Oh well.

 

Water level correction:

This is an odd one. While you are all well aware that water usually flows downhill, rivers in CM are actually lakes, i.e. every square belonging to the same water body has the same water level. In reality, this is of course not the case. But on the map I am currently experimenting with, the difference in elevation along the river is roughly 6m. This leads to the river looking fine at one side:

mg]https://i.imgur.com/DNrYDfO.png[/img]

And being in a deep ravine on the other side:

Water level correction is still a bit experimental but it should in principle get rid of this. You provide the lowest point of the river on the map (UTM coordinates) and the highest point and the tool will calculate a calculate a plain and lower the higher side (and all the other terrain accordingly) to the level of the lower side.

Contour Lines:

@Bulletpoint

I added this option because there were concerns about performance and stair-step effects. And of course it takes much less time to only set contour lines. I tested versions and could not observe any difference in performance. And for stair-step effects...

Every point set:

5m contours:

10m

Doesn't look precisely stair-step effect free to me... Am I doing something wrong?

Ah, one final thing: I changed the license under which the code is published. I can't really see how this affects you but I don't want anyone thinking I'm sneaking an evil ninja license in . You can do whatever you want with the code, you can even sell maps you made with it (as long as BF is ok with it, that is). I only speak broken legalese but I thought the new license might be a bit more fitting for community made stuff: the only restriction is, if you modify the code and make something new with it, you have to make it open source just like the original one.

Right, have fun making maps!

 

.

[Butschi]

And since I've already shown you a bit of the map, here's a litter teaser:

No new features, I just like the autumn looks on this map. Btw. this where I grew up. Well, at the base of the hill in the background in the middle.  

Edited October 25, 2022 by Butschi

[Lucky_Strike]

1 hour ago, Butschi said:

No new features, I just like the autumn looks on this map. Btw. this where I grew up. Well, at the base of the hill in the background in the middle.  

Thanks so much Butschi.

Map pic looks great, does it look familiar to you? Does it look like home?