CMBN weapons effect tests

[Drifter Man]

Over the past few weeks I've been extensively testing the firepower of different American, British and German rifles, SMGs and LMGs in CMBN 4.0. I originally wanted to make a complete series of tests before 'publishing' the results, but I understand that there is an update in the making that will, as a minimum, adjust the rate of fire of some weapons, so for my work to have any impact, I should show them now rather than later. I am posting in general CM discussion. Although the tests were CMBN-specific, I believe that the same patterns in weapons effectiveness will be found in other titles as well.

Method

13 lanes, target troops in foxholes (4-man U.S. medium mortar ammo bearer teams, regular, fanatic, no ammo). Walls are used to separate the lanes over the last 40 meters only, to prevent ricochets from affecting the results. In each lane, one firing team (regular, normal motivation, no leadership modifier) engages the target troop unit using a target arc order. The firing team is so set up that there is only one man in the team with the tested weapon, and he is the only one firing. Ample ammo is provided from supply trucks so the firing unit does not run out of ammunition during the test. There always is a leader in the team with binoculars to aid with spotting at long distances. 13 firing teams with the same weapon are tested in parallel at distances from 40 to 600 meters. The test runs for 10 minutes or until all target troops are eliminated, whichever comes earlier. The total time in action for the 13 teams (between 0 and 7800 seconds) and the total casualties (between 0 and 52) is evaluated. Each test is repeated 26 times, therefore each weapon is tested 338 times at each distance.

An example of the test file is found here: https://www.dropbox.com/s/d5b7qoex6cmigm0/Weapons effects BAR (Gunner) 400m.bts?dl=0

In total, there are 106 files like that one, each run 26 times for up to 10 minutes. About 10,250,000 rounds were fired by the firing teams and 72,530 casualties taken by the target troops.

Distance to target is the average distance of the 13 firing teams to the target action spot (as the firing team's action spot is 8 m wide, this needs to be adjusted for)

Firepower is here defined as the average number of casualties suffered per unit time by the target troops. Minor wounds do not count. The unit is bpm, "bodies per minute".

Rate of fire is taken from ammo consumption during the test, averaged from 26 instances. The unit is rpm, rounds per minute.

Accuracy is the number of casualties per round fired. It may not fully correspond to the number of hits per round fired as one casualty may receive multiple hits, especially with automatic fire. The unit is bpr, "bodies per round", multiplied by 1000 to make the numbers easier to interpret.

Tested weapons

• MP40 (Leader), Sten Mk II (Leader), Sten Mk IV (Leader, Soldier), M1A1 Thompson (Leader) - distances 40-192 m
• MP44 (Leader), MP44 (Soldier) - distances 40-320 m
• Karabiner 98K, Gewehr 43, Lee Enfield No 4, M1 Garand (all Soldier) - distances 40-320 m
• Lee Enfield w/scope (Marksman) - distances 40-600 m
• MG42 LMG, Bren, B.A.R. (all Gunner) - distances 40-600 m

[I am going to break up this post here, results come next]

[Drifter Man]

First of all, some more notes:

• there is no discernible difference in MP44 and Sten effectiveness whether it is fired by a Leader or a Soldier. I am showing Leader data only.
• similarly, there is no conclusive evidence that Sten Mk II and Mk IV are any different. I am showing Sten Mk II data only.
• MP44, Bren and B.A.R. switch from full auto to semi-auto fire above 150 meters, so there is a step change in the results between 120 and 160 meters where I ran the tests
• the statistics may still not be good enough for weapons achieving few kills at long distances, so the results for rifles at the edge of their range are indicative rather than accurate.

Firepower

First note: the vertical axis is in log scale, otherwise the drop in firepower with distance would drown out all detail. I'll post the excel file so you can make any graphs you want.

Bolt-action rifles are at the bottom, Lee-Enfield appears to be superior to Kar 98K. Semi-auto rifles are better, both M1 Garand and Gewehr 43 are on the same level together with the MP44 (when fired in semi-auto mode). All SMGs are better than all rifles over their entire range up to 200 m. MP40 and Sten have very similar performance, Thompson is the most powerful SMG. MP44 is less powerful than SMGs but is in the same league with them until the 150 m mark. Bren and B.A.R. are generally in the league with SMGs as well but, of course, keep going beyond 200 m. The B.A.R. is inferior to the Bren, apparently due to its smaller magazine, and therefore lower average rate of fire. MG42 beats everything by a wide margin except the sniper rifle. The scoped rifle benefits much less from closing the range than the other weapons.

The only problem from my perspective is that SMGs keep their high performance out to 200 m. Their firepower does not fall much with distance in the outer part of their range, and as we will see, their accuracy (on "bodies per round" basis) is constant or even rises between 120 and 200 m.

[more to come]

Edited January 6, 2018 by Drifter Man

[Drifter Man]

Rate of fire

Not that much to see here, bolt-action rifles have the lowest rate of fire (including the scoped Lee-Enfield), semi-auto rifles have a higher one. You can see the difference between the Bren and B.A.R. as a result of magazine capacity (30 vs 20 rounds). MG42 has about 2-6 times higher RoF than everything else.

Accuracy

I want to highlight again that we are talking about kills per round, not about hits per round.

The green line on the top is the scoped Lee-Enfield. It's accuracy is super-high but does not increase that much with decreasing distance, so there is little advantage in getting close. Bolt-action rifles are more accurate than semi-auto rifles, although this cannot be confirmed at long distances (sample size issue, I believe). Bren and B.A.R. achieve more kills per round than over 240 m. The SMGs are at the bottom but as you can see, their accuracy is flat or even rises with distance above 120 m!

Excel file link (individual data is on hidden sheets if you need to see them): https://www.dropbox.com/s/qeltpvi732w91bt/CMBN weapons effects.xlsx?dl=0

[weapon2010]

very interesting data

[Bulletpoint]

Very impressive. I think there's a lot to discuss in this, and hopefully improve upon. Thanks for doing all this work.

One thing about the methodology - you said each firing team only consists of one guy. But also you said that there's a spotter with binoculars to aid with spotting at a distance. If you mean he is in a separate team, I don't think he affects the spotting of the firing teams.

[akd]

Huge props to you for controlling variables and achieving a meaningful sample size.  I have an issue with your criteria for "firepower."  I think it is an oversimplication of the number of factors that go into firepower, leading to a mischaracterization of the scoped rifles having the greatest "firepower."  Firepower is really a combination of hit rate, ability to penetrate cover and ability to generate suppression.  For example, put an MG team in a building then hide it after it is initially spotted.  Do the scoped rifles still have the greatest "firepower" to neutralize this threat?

[Bulletpoint]

Is there any reason to believe the MP44 would be less deadly at close range than the MP40, the Thompson, etc.?

[Erwin]

Is it therefore a historical misperception that the MP44 "assault rifle" was greatly superior to bother the MP40 "machine pistol" and the Sten?  Or, does the game simply misrepresent them?  And since it is a game, does it really matter?  Wondering if doing this sort of analysis of an entertainment product is a bit pointless and mitigates the immersive verisimilitude. 

[Bulletpoint]

2 minutes ago, Erwin said:

And since it is a game, does it really matter?  Wondering if doing this sort of analysis of an entertainment product is a bit pointless and mitigates the immersive verisimilitude. 

I think it greatly matters. For me, a big part of the enjoyment of this game is that it strives to be realistic, both about weapons and tactics.

If I just wanted to see some shooting and big explosions, there would be at least 20 others games I could play that had way better graphics.

[Bulletpoint]

I've been thinking about why the SMGs suddenly seem to become more deadly at longer ranges.

Could it be because of plunging fire?

You said you put the target squads in foxholes, so since the low velocity bullets fly in more of an arc trajectory, maybe they would have a tendency to drop into the foxholes from above?

Not sure how foxholes are modelled technically..

[Erwin]

11 minutes ago, Bulletpoint said:

I think it greatly matters. For me, a big part of the enjoyment of this game is that it strives to be realistic, both about weapons and tactics.

Agreed.  But when we all assumed everything was accurate, the game was great.  Now, we find out that it's not so accurate - and it's starting to bother (some of) us.  It's pointless.  Better to not know this stuff is what I am saying.

[Bulletpoint]

2 minutes ago, Erwin said:

Agreed.  But when we all assumed everything was accurate, the game was great.  Now, we find out that it's not so accurate - and it's starting to bother (some of) us.  It's pointless.  Better to not know this stuff is what I am saying.

I think it's still great. But while I'm no WW2 ballistics expert, I think these tests show some things could be improved. I just like to see a good thing get better. So I think it's very nice people are putting their minds to help improve it.

[Erwin]

Lots of things need to be improved.  This doesn't improve anything.  It simply puts yet more items for people to complain about and on BF's "to do" list - and they are probably overloaded as it is.  Have to be cognizant of priorities.  The game will never be perfect, there will always be compromises.  In terms of creating a fun and rewarding wargame experience, verisimilitude is much more important than getting technical physics aspects accurate.  

[A Canadian Cat]

43 minutes ago, Bulletpoint said:

I just like to see a good thing get better. So I think it's very nice people are putting their minds to help improve it.

+1 to that. Don't listen to the glass half empty, sky is falling or world is coming to an end crowd. Most of us *do* want to see things get better / fixed.

 

[c3k]

I haven't looked at anything other than what is posted, above.

If you are counting "bodies per minute", then the reload rate from the ammo truck would make a difference, yes? 

Does an MG42 get a 200 round belt, fire it off, then get another? Does a US rifleman get an 8 round clip for his Garand? Do they have the same delay between reloads from the truck?

Just a minor quibble. Or question.

Gotta say I like the presentation. Plenty to chew upon.

Thanks.

[Erwin]

"Most of us *do* want to see things get better / fixed."

Of course we do.  But there is so much.  Need to prioritize what can be accomplished with available resources is what am saying.  Worrying about such detail is akin to worrying about how many rivets are on the underside of the Pz Mk xyzv4.

Edited January 6, 2018 by Erwin

[Bulletpoint]

1 hour ago, Erwin said:

Lots of things need to be improved.  This doesn't improve anything.  It simply puts yet more items for people to complain about and on BF's "to do" list - and they are probably overloaded as it is.  Have to be cognizant of priorities. 

You could also look it from the opposite direction: If Battlefront had to do this whole enormous testing suite and statistical analysis, it would have cost them a lot of time - or a lot of money, if they hired somebody to do it.

Now the results are there for the taking, free of charge. If they want to adjust weapon parameters they now have a good starting point.

If they don't want to do it, they are going to spend exactly zero resources on this thread. I'm sure they are fully able to prioritise their own time

Quote

Worrying about such detail is akin to worrying about how many rivets are on the underside of the Pz Mk xyzv4.

I disagree. It's not a cosmetic thing at all. It is relevant to the basic gameplay, and the same improvements would improve several titles, since many of the same weapons are used across the games.

 

Edited January 6, 2018 by Bulletpoint

[Ts4EVER]

3 hours ago, Erwin said:

Is it therefore a historical misperception that the MP44 "assault rifle" was greatly superior to bother the MP40 "machine pistol" and the Sten?  Or, does the game simply misrepresent them?  And since it is a game, does it really matter?  Wondering if doing this sort of analysis of an entertainment product is a bit pointless and mitigates the immersive verisimilitude. 

The MP44 is not better than a submachine gun at close range or a semi-automatic full caliber rifle like the Garand at medium range. However, the point is that it is capable of doing both. You can deliver reasonably accurate aimed fire over distance or jump into a trench and mow down some people with the same rifle.

[DMS]

7,92x33 has higher recoil. That's why pistol round it better <100 metres. 7,62x25 - <200 metres.

[Drifter Man]

6 hours ago, Bulletpoint said:

Very impressive. I think there's a lot to discuss in this, and hopefully improve upon. Thanks for doing all this work.

One thing about the methodology - you said each firing team only consists of one guy. But also you said that there's a spotter with binoculars to aid with spotting at a distance. If you mean he is in a separate team, I don't think he affects the spotting of the firing teams.

There was only one man on the team shooting, but he had teammates with weapons using different ammunition the team didn't have, so they didn't shoot. The leader with binoculars was part of the team For example:

1. Split off an assault team from an understrength U.S. Rifle Squad - Leader (Thompson) + Soldier (M1 Garand)
2. Give them an area target order so that they run out of ammo (set supply to scarce in the editor to speed up things)
3. Reload either .45 cal from trucks if you want to test the Leader with the Thompson, or .30 cal M2 if you want to test the Soldier with the Garand, but make sure the other guy has no ammo
4. Repeat 13 times and you have your firing teams.

[Drifter Man]

6 hours ago, akd said:

Huge props to you for controlling variables and achieving a meaningful sample size.  I have an issue with your criteria for "firepower."  I think it is an oversimplication of the number of factors that go into firepower, leading to a mischaracterization of the scoped rifles having the greatest "firepower."  Firepower is really a combination of hit rate, ability to penetrate cover and ability to generate suppression.  For example, put an MG team in a building then hide it after it is initially spotted.  Do the scoped rifles still have the greatest "firepower" to neutralize this threat?

I agree and I should have included that in my comments. The parameter I measure is not a complete picture of firepower because it does not include suppression. Suppression could be also measured with my method but it would be incredibly time-consuming. Sniper rifle is exactly the example where suppression is completely out of match with the ability to hit and kill. Maybe "killing power" or something like that would be a better term.

[Drifter Man]

4 hours ago, Erwin said:

Is it therefore a historical misperception that the MP44 "assault rifle" was greatly superior to bother the MP40 "machine pistol" and the Sten?  Or, does the game simply misrepresent them?  And since it is a game, does it really matter?  Wondering if doing this sort of analysis of an entertainment product is a bit pointless and mitigates the immersive verisimilitude. 

My original motivation was to find out how different weapons work, to be able to select and use my forces better. Based on the data, I personally have no reason to think that anything is wrong, except maybe the effectiveness of SMGs not decreasing with distance at longer ranges. If this is intended behavior, I'm fine with it. If the development team is not aware of it and sees it as a problem, this may help them.

About the MP44, I think it's like what @Ts4EVER said - can be used like a SMG at close range and like a rifle at medium range, and it can do both. I wouldn't change a thing about how it is represented in the game.

[Drifter Man]

5 hours ago, Bulletpoint said:

I've been thinking about why the SMGs suddenly seem to become more deadly at longer ranges.

Could it be because of plunging fire?

You said you put the target squads in foxholes, so since the low velocity bullets fly in more of an arc trajectory, maybe they would have a tendency to drop into the foxholes from above?

Not sure how foxholes are modelled technically..

I'm not sure either, but I suspect that foxholes just provide a "cover modifier" that converts fewer hits into kills. I've seen troops regularly shoot through one foxhole to hit a man cowering in another foxhole behind it.

I put the target troops in foxholes to get meaningful close-range results (otherwise it would be just a massacre), but it may have introduced other uncontrolled variables.

Edited January 6, 2018 by Drifter Man

[Drifter Man]

3 hours ago, c3k said:

I haven't looked at anything other than what is posted, above.

If you are counting "bodies per minute", then the reload rate from the ammo truck would make a difference, yes? 

Does an MG42 get a 200 round belt, fire it off, then get another? Does a US rifleman get an 8 round clip for his Garand? Do they have the same delay between reloads from the truck?

Just a minor quibble. Or question.

Gotta say I like the presentation. Plenty to chew upon.

Thanks.

I reloaded them from the truck before the test, so they carried all their ammo with them. 1000 rounds for rifles, 2000-3000 for SMGs, 3000-4000 for MGs. In all cases it took about 7-8 seconds to reload any weapon in the test. HMGs need a bit more time (about 15 seconds I think) but I didn't get to testing HMGs.

[c3k]

26 minutes ago, Drifter Man said:

I reloaded them from the truck before the test, so they carried all their ammo with them. 1000 rounds for rifles, 2000-3000 for SMGs, 3000-4000 for MGs. In all cases it took about 7-8 seconds to reload any weapon in the test. HMGs need a bit more time (about 15 seconds I think) but I didn't get to testing HMGs.

Perfect. I misunderstood.