Before we start you can go ahead and delete your WasDirt command blocks at the end (the last two). 
 

Ok the first thing we will be doing is making the armor stand a bit more lag friendly, not reducing much lag but incase your game is lagging these command blocks will help out. It also makes it easier especially with this new system.

 

We will change the first command to execute as @e[tag=RightClickDemo] at @s unless entity @e[type=player,tag=Dirt,distance=..1] run kill @s. This command is multiplayer compatible, all it does it make sure there are none of the invisible armor stands that aren't near a player holding dirt as that could take their items. It should be a fairly simple command so we will move on without much explanation.

The next two commands will remain the same. The fourth command will be changed slightly too execute at @e[tag=Dirt] unless entity @e[type=armor_stand,distance=..1] run summon armor_stand ~ ~ ~ {ShowArms:1b,NoGravity:1b,Invulnerable:1b,Invisible:1b,Tags:["RightClickDemo"]}. This should be the same except we added the unless entity part. Basically if there is a armor_stand close to you it won't make a new one it will just teleport that one to you. THe teleport part is the next Command.

Here we will insert a new command inbetween the last one and the fifth one from before. execute at @a[tag=Dirt] if entity @e[type=armor_stand,distance=..1] run tp @e[type=armor_stand,tag=RightClickDemo,sort=nearest,limit=1] @p. This command is the tp command, you should be able to digest this without an explanation so we will move on.

​

The next command (what was the fifth but will now be the sixth) will be modified slightly to execute as @e[tag=Dirt] at @s if entity @e[type=armor_stand,tag=RightClickDemodistance=..1,nbt={HandItems:[{id:"minecraft:dirt",Count:1b},{}]}] run give @s diamond. We added the if entity part. Not only do we check if you got rid of the dirt but also if there is an Right Click armor stand within one block that has a dirt item. Now since the armor stand will have your dirt and won't be able to work anymore we want to make sure it dies straightaway because otherwise you can't use another item until you move away. execute as @e[tag=WasDirt] at @s as @e[type=armor_stand,distance=..1,nbt={HandItems:[{id:"minecraft:dirt",Count:1b},{}]}] run kill @s is the command. It is the same thing except instead of giving you a diamond or whatever you want it to do it kills the armor stand, also instead of if entity we use as. This will be after all your other command blocks if you have multiple for the functionality part. Also, NBT is pretty complicated so if the formatting is weird don't worry. Personally even though I know most of nbt by now I still use generators just in case. I will recommend the same generator that I use.