Possibly the term 'armies' makes it sound much a bigger issue than was actually the case. By 1943 there was an undeniable proliferation of special units in the Med, most coming from units that had been formed in the Middle East/North Africa. Quick memory test -
Special Air Service
Special Boat Section, later Squadron
Demolition Squadron (aka Popski's Private Army)
Long Range Desert Group
Special Raiding Squadron
Raiding Support Regiment
Those are the ones I can recall offhand, sure I've missed a couple or three. While the list is quite long some of these units measured their strength in the few hundreds, so without going through the numbers in detail I reckon you'd find enough personnel from them all to fill out a single Infantry Brigade, or roughly 2500 men.
A constant argument against special forces was that they deprived the infantry especially of small unit leaders, Corporals and Serjeants, and therefore reduced the effectiveness of the infantry arm. The same criticism though was levelled at the Special Service (later Commando) Brigades, and to an extent also the Airborne Divisions. Airborne and Commando units and formations took a decent chunk of those men who might otherwise have found themselves in the infantry proper, and because their standards were more exacting and the Commandos were entirely volunteer (and selective), they tended to pick the best. But that's no different than what happened with German and US Airborne forces for example, though the British did sink a lot more resources into special units than I think anyone.
As to whether the large number of special units made a difference, that's probably down to individual interpretation. Generals commanding Corps and Armies probably think what they're doing is having the most direct impact on the campaign, and as ever want more personnel to increase that impact. Special forces advocates might offer that their activities are forcing the enemy to divert far more of his men to rear area security, large search and sweep operations to try and locate small teams of men, and in so doing doing are ensuring said Generals are facing fewer enemy troops. Also sustaining and operating large conventional formations in places like occupied Greece or the Balkans wasn't exactly a viable proposition, whereas as small specialised units, often working with resistance networks, proved doable. Whether it was worth the cost, again various views to be found.