This is just a Mathematica feature.
I.e., if you use -> (Rule) the right-hand side is evaluated before
applying the rule (and Tr[a.b.GA[5]] just evaluates to 0).
So, just use :> (RuleDelayed):
(Please, to all: study and learn Mathematica. There are screencasts,
coursese etc.)
In[1]:= <<HighEnergyPhysics`fc`
Loading FeynCalc from /home/rolfm/HighEnergyPhysics
FeynCalc 8.0.0.beta2 Type ?FeynCalc for help or visit http://www.feyncalc.org/
$PrePrint is set to FeynCalcForm. Use FI and FC to change the display format.
Loading FeynArts, see www.feynarts.de for documentation
FeynArts 3.4 patched for use with FeynCalc
In[2]:= expr = GS[a, b, c, d];
In[3]:= expr /. Dot[x_, y__] :> Tr[Dot[x,y,GA[5]]]
Out[3]= (-4 I) eps[a, b, c, d]
This archive was generated by hypermail 2b29 : 09/04/20-12:55:05 AM Z CEST