Kohlenklau Test Dummy Log for learning Butschi's New Map Elevation Tool

[kohlenklau]

I normally use Google Earth Pro in my CM scenario map preparations. It has some type of elevation data down in the lower right. It seems to be the ground. In GE, as I move the cursor along streets and then up over a building, I do NOT see it seem to change to represent the top of a large office building. I could be wrong.

I signed up for a username in USGS Earth Explorer and can get 3 different formats for DEM of a targeted area. *.bil,  *.tif and *.dt2 file formats.

In USGS EE, you zoom in with a map and define the area. It doesn't seem to allow a fine fine tuning to the tiny maps I want. I am getting files of about 25MB.

I found a free converter to make the tif a csv but it has a 5MB max size.

I also signed up for a free course in ARCGIS. It comes with a user license for using ARCGIS online for 6 weeks. It is very powerful to manipulate these GIS related files.

I am stuck but will not give up.

Thank you Herr Doktor!

@Butschi

Edited December 13, 2022 by kohlenklau

[Butschi]

25 minutes ago, kohlenklau said:

OK, I successfully got it to work.

Oh boy. The 5 seconds got me the first try as I had the forum window open and alt-tab did not get me straight to CMFI in time!

I then closed the internet browser. Now I only had python command window and the CMFI game so alt-tab for my IBM PC got me over to see the big show!

Of course I had to re-enter the command line.

I did abort the process before completion. It was already my morning, I had my coffee already in my cup.

But if I had started last night then my advertising slogan (revised with just saying completed elevations versus "completed map") would have been true...

Now I must go and focus on where I get my desired DEM data and how to custom make the input csv file. 

"CMAutoEditor works while you sleep and dream of your completed elevations served with coffee in the morning!"

Awesome!

Ok, it would be easy to increase the countdown it or make it configurable. I'll look at it tonight.

[Butschi]

16 minutes ago, kohlenklau said:

I normally use Google Earth Pro in my CM scenario map preparations. It has some type of elevation data down in the lower right. It seems to be the ground. In GE, as I move the cursor along streets and then up over a building, I do NOT see it seem to change to represent the top of a large office building. I could be wrong.

I signed up for a username in USGS Earth Explorer and can get 3 different formats for DEM of a targeted area. *.bil,  *.tif and *.dt2 file formats.

In USGS EE, you zoom in with a map and define the area. It doesn't seem to allow a fine fine tuning to the tiny maps I want. I am getting files of about 25MB.

I found a free converter to make the tif a csv but it has a 5MB max size.

I also signed up for a free course in ARCGIS. It comes with a user license for using ARCGIS online for 6 weeks. It is very powerful to manipulate these GIS related files.

I am stuck but will not give up.

Thank you Herr Doktor!

@Butschi

I'll share my tool to convert geo-tiff files to csv. With that you can at least try it the data is good enough for you.

Also, I never tried ArcGIS but QGIS is open source and free and also quite powerful.

[kohlenklau]

7 minutes ago, Butschi said:

I'll share my tool to convert geo-tiff files to csv. With that you can at least try it the data is good enough for you.

Also, I never tried ArcGIS but QGIS is open source and free and also quite powerful.

That would be awesome. Thanks

QGIS. great. I had not yet put down any money so I am glad it is free versus ARCGIS.

I think in our PM I sent you a dropbox link for the geotiff for Krasny Bor, Russia.

I don't yet know what is good enough data for making a CM map. 

I think QGIS would have tools such as interpolation to fill in areas between the raw data points. If needed by CM. I guess. 

[kohlenklau]

CMAutoEditor for Dummies

(Revision 13Dec2022)

STEP 1:

In this step you will go download a few things and put them in the best place.

You need some python type program and if you never use python then it is suggested to get miniconda3 as of 13Dec2022.

Go to conda.io and pick the proper OS and download to an easy to find location in your main drive. I used C:\Users\Phil

I had an issue with being an old cheap baztard and still use Windows 7 and had to go to the archives to get an older version of miniconda3 that worked with Win7.

For the Windows 7 issue, here is the "archive" for finding the older versions to download.

https://repo.anaconda.com/miniconda/

I ended up using Miniconda3-py37_4.8.2-Windows-x86_64 dot exe

About 52MB application file

It was not zipped and after I downloaded it, I just pasted it over to C:\Users\Phil

STEP 2:

Then you get Butschi's actual python code from github at his link. As of 13Dec2022, it was at https://github.com/DerButschi/CMAutoEditor

Tip: if you just want the code and not be watching all the bug tracking, hit the green button

It is a zip file

I also put his folder in C:\Users\Phil. Note: it was a folder in a folder so I had to cut the inner folder and paste to C:\Users\Phil. Follow me?

for my test of the elevation only version of CMAutoEditor the folder was called "CMAutoEditor-master"

so, go to C:\Users\Phil and shift-right click to see a window with an option for "Open command window here"

You get the black command window with C:\Users\Phil>

The command window opened and I successfully changed directory and installed the requirements.

cd C:\Users\Phil\CMAutoEditor-master

then it shifts directory and you see C:\Users\Phil\CMAutoEditor-master>

then type pip install -r requirements.txt

It scrolls a bunch of code stuff as it loads things

It is done just once and you never do it again as long as you don't update CMAutoEditor (in which case the requirements file may change so you would have to run pip install -r requirements.txt again)... BUT if Butschi makes improvements and this becomes an ongoing developing tool, you will want to do it (pip install -r requirements.txt) again, he hopes.

STEP 3:

In this step you will go prepare to actually fire up "old Bessy"...

Maybe you had closed the python command window, maybe you did not...

Start your CM game you want the map to be in. Go to the CM Editor, go to map, go to "Elevation" menu and click "Direct". There is no need to do other settings before that and you don't have to save anything. Make sure that the starting map is 320m x 320m. Which is default for new map anyway!

Note: this is just the settings for CMAutoEditor to start the map which can be bigger. You don't dial up your start map to be bigger.

Close out of your other windows and such I think.

STEP 4:

Just have CM and the black command window.

If you had closed the black command window, do this:

Go to your equivalent location of C:\Users\Phil

shift-right click to see a window with an option for "Open command window here"

You get the black command window with C:\Users\Phil>

Then type cd C:\Users\Phil\CMAutoEditor-master

then you see C:\Users\Phil\CMAutoEditor-master>

then type all as one command python .\cmautoeditor.py -i example_height_map.csv

(one command)

(Note: the csv file name will obviously change as you use YOUR data set. This csv file should be in the folder with CMAutoEditor)

Then the 5 second countdown starts. ALT-TAB TO CM Window. MACHT SCHNELL!!!

Don't touch your mouse.

It starts doing its thing as shown in Butschi's youtube video posted down at the bottom. 

It adjusts map size as it does its thing. You can watch and see it cranking away.

There is currently no window or message to say "I am done"...

"CMAutoEditor works while you sleep and dream of your completed elevations served with coffee in the morning!"

 

Edited December 13, 2022 by kohlenklau

[Butschi]

[Vacillator]

I'm loving what you guys are doing, good luck amigos!

[Butschi]

New release:

1.1.0

- New script: geotiff2cm.py. Converts geotiff files into .csv files usable by CMAutoEditor.
- Quality of Life update: Added command line argument to set countdown (usage: --countdown 10)

geotiff2cm.py:

--input-file: Select input geotiff
--output-file: Set name of output file (will have csv format)
--bounding-box: Set a box in which data is extracted. Some geotiffs are really large and you should always set a box. Usage: EPSG no., left, bottom, right, top (e.g. 23700, 647058.157, 240190.841, 647609.303, 240446.250)

Note: For epsg numbers see e.g. here: https://epsg.io/
They are used to specify the meaning of the coordinates of the box, i.e. the coordinate reference system.

https://github.com/DerButschi/CMAutoEditor/releases/tag/1.1.0

EDIT: This is where you want to do pip install -r requirements.txt after updating the code.

Edited December 14, 2022 by Butschi

[Butschi]

9 hours ago, kohlenklau said:

Then the 5 second countdown starts. ALT-TAB TO CM Window. MACHT SCHNELL!!!

Fixed that for you.

[Aragorn2002]

Thanks, guys, much appreciated. Gonna give it a go. I will keep you informed.

So there's no limit to the size of the map?

Edited December 14, 2022 by Aragorn2002

[Butschi]

31 minutes ago, Aragorn2002 said:

So there's no limit to the size of the map?

Only the usual limits imposed by the CM scenario editor if that's what you mean?

[Aragorn2002]

3 minutes ago, Butschi said:

Only the usual limits imposed by the CM scenario editor if that's what you mean?

It is. Thanks, Butschi. And another question (remember I haven't read the instructions yet, will do so today). The elevation data is from Google Earth or another source?

Edited December 14, 2022 by Aragorn2002

[Butschi]

7 hours ago, Butschi said:

--bounding-box: Set a box in which data is extracted. Some geotiffs are really large and you should always set a box. Usage: EPSG no., left, bottom, right, top (e.g. 23700, 647058.157, 240190.841, 647609.303, 240446.250)

I was in a bit of a hurry to get some sleep, so I left this point a little... underexplained. Those geotiffs often span a rather large area, often thousands of square kilometers! So, in most cases you will want to extract on a (small) part of the data. Now, how do we that? Basically, you give geotiff2cm.py a lower left (left, bottom) coordinate and an upper right (right, top) coordinate of the box from which you want the data.

Here is where it gets a little complicated. We must specify what those coordinates mean. Let me elaborate on this a bit. Unless you are a flat-earther, you are aware that our planet is a sphere. Maps, on the other hand, are often flat. So maps are usually a projection of that sphere onto some plain. Thing is, there is a lot of different ways how to do this and the plain itself is different depending on where on earth you are (and different institutions use different set of coordinates, etc.). Even the sphere itself that was used for calculating differs. So, we have to specify which set of coordinates, either spherical in degrees longitude/latitude or projected in meters or whatever we mean. This is where this epsg code comes in. We can go into more details in case you want to work with QGIS or ArcGIS or similar but let's just do the most simple version of getting your coordinates:

Go to google maps, left click on a point on the map and google will give you two numbers which is degrees latitude and longitude. Caution, for some reason coordinates are often given that way but it should be longitude, latitude as longitude is more or less the x coordinate and latitude the y coordinate. So, take two of those points as your lower left and upper right coordinate. Now just use 4326 as epsg number (this means degrees longitude/latitude in WGS84, which is the most used spherical coordinate reference system).

This should do the trick. Write down your set of coordinates somewhere as you might need them at a later map making stage.

[Butschi]

24 minutes ago, Aragorn2002 said:

It is. Thanks, Butschi. And another question (remember I haven't read the instructions yet, will do so today). The elevation data is from Google Earth or another source?

I don't know what @kohlenklau's status on extracting data from google earth is. Alas, generally there doesn't seem to be a one-size-fits-it-all solution around, as globally available satellite data is usually just the raw elevation, including tree tops, buildings, etc. But there are often good national sources of data around are often free (though not always). Just make sure you get a digital terrain model, not a digital surface model (because that is the one with tree tops, buildings, ...).

In your case, I think you were interested in elevation data from Poland? You can get geotiffs here: https://www.geoportal.gov.pl/uslugi/usluga-sieciowa-wcs

Maybe we should make a list with good sources?

[Aragorn2002]

15 minutes ago, Butschi said:

I don't know what @kohlenklau's status on extracting data from google earth is. Alas, generally there doesn't seem to be a one-size-fits-it-all solution around, as globally available satellite data is usually just the raw elevation, including tree tops, buildings, etc. But there are often good national sources of data around are often free (though not always). Just make sure you get a digital terrain model, not a digital surface model (because that is the one with tree tops, buildings, ...).

In your case, I think you were interested in elevation data from Poland? You can get geotiffs here: https://www.geoportal.gov.pl/uslugi/usluga-sieciowa-wcs

Maybe we should make a list with good sources?

Yes, I'm especially interested in former East Prussia and the towns of Goldap and Schirwindt in particular. 

A list with good sources would be very helpful indeed.

[kohlenklau]

I messaged Butschi and told him I will continue this thread with a few real world examples of the entire process. So, we stay in learning mode and I am sure Herr Doktor will have to enlighten us as I proceed. I will stab at 4 cases simultaneously! Insane but why not?.

CASE 1 (Buda, Hungary)

Butschi has kindly obtained for me (3 Euros) from the Hungarians, the data for an area of Buda, Hungary I wanted to "Butschify" into a CMRT map.

Yes, some of the nations have the data for free, some charge for shipping & handling. 

CASE 2 (Goldap, former East Prussia- now in Poland)

elevation data from Poland? You can get geotiffs here: https://www.geoportal.gov.pl/uslugi/usluga-sieciowa-wcs

CASE 3 (Krasny Bor, Russia)

CASE 4 (Kasserine Pass, Tunisia)

[kohlenklau]

CASE 1 (Buda, Hungary)

Butschi has kindly obtained for me (3 Euros) from the Hungarians, the data for an area of Buda, Hungary I wanted to "Butschify" into a CMRT map.

It ran very quickly! A small Kohlenklau-sized map of about 544 x 256. Like 5 minutes. Holy moley that is fast. 

Unfortunately it was not the exact area I wanted so Herr Doktor will have to teach me (and you) the way to select the exact corner points and MAKE it the desired area perfectly. Ideally this CMAutoEditor tool is used "up early in the scenario and map development" phase to get the lay of the land perfect. THEN you can add buildings and streets and foliage. 

 

[kohlenklau]

4 hours ago, kohlenklau said:

A small Kohlenklau-sized map of about 544 x 256. Like 5 minutes. Holy moley that is fast. 

@Butschi You got the 1km x 1km (1 sq km) but then you did the corner plots to define it smaller? Or did I maybe quit too early while she was doing her thing? I waited and it seemed non-active at 544x256. If you DID do the corner point defining then maybe I just need to look at the sq km of Buda you bought me for 3 Euro. Real estate is a good investment! 

[Butschi]

26 minutes ago, kohlenklau said:

@Butschi You got the 1km x 1km (1 sq km) but then you did the corner plots to define it smaller? Or did I maybe quit too early while she was doing her thing? I waited and it seemed non-active at 544x256. If you DID do the corner point defining then maybe I just need to look at the sq km of Buda you bought me for 3 Euro. Real estate is a good investment! 

I just more or less randomly chose two coordinates and extracted a smaller box of that 1 sq km. Just for demonstration.

You can see the full area here:

You can do the visualization of this with QGIS e.g. The nifty thing about geotiffs is that they contain additional data about where the area is, how it is oriented, etc. If you look closely, it is not exactly a square which is why you really should not take the whole area.

Getting the coordinates for a box is really easy, as I described above. You can just do two clicks in google maps.

The full command then looks like

python geotiff2cm.py --input-file merged_ddm.tif --bounding-box 4326, 19.008865, 47.504562, 19.016790, 47.508557 --output-file buda.csv

Coordinates are just an example.

 

Edited December 14, 2022 by Butschi

[kohlenklau]

1 hour ago, Butschi said:

python geotiff2cm.py --input-file merged_ddm.tif --bounding-box 4326, 19.008865, 47.504562, 19.016790, 47.508557 --output-file buda.csv

oh, now I see. I will go and find my 2 points. I have a nice break starting on Monday so this is something to tackle over the holidays. 

[kohlenklau]

My goal for today is to try and create a step by step written guide for SUCCESSFULLY taking Google Earth data and ending up with the proper CVS format to feed into CMAutoEditor!

This COULD be a big flop. We'll have to see.

Step 1 is the CM scenario/map creator opening Google Earth Pro (GEP) to find his area of interest (AOI) for the scenario.

I use a few things I have stumbled upon. Sometimes I use only part of their process and then have to take a detour.

The below video does cover the use of Google Earth Pro to get points in the AOI using the path tool. This set of points is saved as a *.kml file

Here is a link to the one tool that he uses to convert/expand *.kml to *.gpx

https://www.gpsvisualizer.com/elevation

But the points dataset obtained by the GEP path tool method needs interpolation to figure the elevation for missing points in a proper grid.

I will now try to harness QGIS freeware (versus paying for ARCGIS) for the interpolation. 

Finally here is a GPX to CSV converter https://mygeodata.cloud/converter/gpx-to-csv

That is my rough plan for today. I will report back.

 

[AlexUK]

This is super interesting. I’m glad I’m not having to scratch my head over this. Excellent work both!!

[kohlenklau]

2 hours ago, AlexUK said:

I’m glad I’m not having to scratch my head over this.

oh dang, I wish I was NOT scratching my head. Hair is gone. Scalp skin gone. Cranium bone almost gone. I better stop before I hit brain tissue...or a hollow void?

 

 

[Butschi]

On 12/13/2022 at 4:01 PM, kohlenklau said:

There is currently no window or message to say "I am done"...

New update 1.2.0:

https://github.com/DerButschi/CMAutoEditor/releases/tag/1.2.0

Small Quality of Life update. You will get an info box once you run cmautoeditor.py:

The countdown will only start once you hit the Ok button. When CMAutoEditor is done processing the data you get an info box:

 

Edited December 15, 2022 by Butschi

[kohlenklau]

1 minute ago, Butschi said:

Small Quality of Life update

That is a great extra feature. Thank you from the non-coder common mortals!